Nov 01 19:58:02.147022 ip-10-0-133-52 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 19:58:02.566220 ip-10-0-133-52 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:58:02.566220 ip-10-0-133-52 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 19:58:02.566220 ip-10-0-133-52 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:58:02.566884 ip-10-0-133-52 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574196    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574295    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574299    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574302    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574305    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574308    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.574301 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574311    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574315    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574318    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574321    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574323    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574329    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574332    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574334    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574337    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574340    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574342    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574345    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574347    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574350    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574352    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574355    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574358    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574360    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574362    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.574552 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574365    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574368    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574370    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574373    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574376    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574378    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574381    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574384    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574386    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574389    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574391    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574394    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574396    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574399    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574402    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574405    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574408    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574410    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574413    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574416    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.575044 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574418    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574421    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574424    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574426    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574428    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574431    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574434    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574437    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574439    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574442    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574444    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574447    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574450    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574452    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574454    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574457    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574460    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574462    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574464    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574467    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.575510 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574469    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574472    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574476    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574484    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574487    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574490    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574493    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574496    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574499    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574501    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574504    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574506    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574509    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574511    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574514    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574517    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574521    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574526    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574530    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.575994 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574533    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574535    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574538    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574540    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574543    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574546    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574548    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574551    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574553    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574556    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574559    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574561    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574564    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574567    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574570    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574573    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574575    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574579    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574582    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574585    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.576444 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.574588    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.577265 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577251    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577266    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577271    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577275    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577278    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577281    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577284    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577286    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577289    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577292    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577295    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577298    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.577297 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577301    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577304    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577307    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577309    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577312    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577314    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577317    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577319    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577322    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577324    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577327    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577330    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577332    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577335    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577337    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577340    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577342    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577345    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577348    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577351    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.577599 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577353    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577355    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577358    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577360    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577363    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577365    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577368    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577370    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577373    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577375    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577378    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577381    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577383    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577386    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577391    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577393    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577396    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577399    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577401    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577404    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.578101 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577406    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577409    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577412    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577414    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577417    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577420    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577422    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577425    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577427    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577430    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577433    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577435    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577438    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577440    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577443    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577445    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577448    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577450    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577453    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.578586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577455    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577457    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577460    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577462    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577465    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577473    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577477    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577480    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577483    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577487    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577490    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577492    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577495    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577497    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577500    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577503    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577505    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577508    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577510    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577513    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.579056 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577516    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577518    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577521    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577523    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577526    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577528    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577531    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577533    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577536    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577538    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577541    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577543    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577546    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577548    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.577553    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578140    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578147    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578152    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578156    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578159    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.579514 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578162    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578165    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578168    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578170    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578173    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578175    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578179    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578183    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578187    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578189    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578192    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578195    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578197    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578200    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578203    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578205    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578208    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578210    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578213    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.580003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578215    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578218    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578220    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578223    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578225    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578228    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578230    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578233    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578236    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578239    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578242    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578245    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578248    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578251    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578254    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578257    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578259    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578262    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578264    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.580445 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578266    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578270    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578274    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578276    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578279    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578281    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578284    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578286    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578290    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578292    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578295    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578297    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578300    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578302    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578305    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578307    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578310    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578313    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578315    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.580951 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578318    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578321    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578323    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578330    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578332    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578337    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578339    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578342    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578345    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578359    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578363    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578366    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578369    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578372    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578374    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578377    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578382    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578385    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.581398 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578387    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578390    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578392    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578395    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578398    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578400    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578403    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578405    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578408    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578410    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578413    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578415    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578418    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578420    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578423    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578426    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578428    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578431    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578433    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578436    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.582016 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578441    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578443    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578446    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.578448    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578551    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578558    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578565    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578570    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578574    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578577    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578582    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578586    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578590    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578593    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578597    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578600    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578603    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578606    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578609    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578612    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578615    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578618    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578622    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 01 19:58:02.582484 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578625    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578628    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578632    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578635    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578639    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578642    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578658    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578661    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578664    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578669    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578672    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578675    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578678    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578683    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578686    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578689    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578692    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578696    2576 flags.go:64] FLAG: --enable-server="true"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578699    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578704    2576 flags.go:64] FLAG: --event-burst="100"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578708    2576 flags.go:64] FLAG: --event-qps="50"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578711    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578714    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578717    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 01 19:58:02.583024 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578721    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578724    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578727    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578730    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578733    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578736    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578739    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578742    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578745    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578748    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578751    2576 flags.go:64] FLAG: --feature-gates=""
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578755    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578758    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578762    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578765    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578769    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578772    2576 flags.go:64] FLAG: --help="false"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578775    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578779    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578782    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578785    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578789    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578792    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 19:58:02.583603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578795    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578798    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578801    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578804    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578808    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578811    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578814    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578817    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578820    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578823    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578826    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578829    2576 flags.go:64] FLAG: --lock-file=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578832    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578835    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578838    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578844    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578847    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578851    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578854    2576 flags.go:64] FLAG: --logging-format="text"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578857    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578860    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578863    2576 flags.go:64] FLAG: --manifest-url=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578866    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578871    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578874    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 19:58:02.584166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578878    2576 flags.go:64] FLAG: --max-pods="110"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578881    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578884    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578888    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578891    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578894    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578897    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578900    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578908    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578911    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578914    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578917    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578920    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578923    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578926    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578929    2576 flags.go:64] FLAG: --port="10250"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578932    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578935    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a6b04bdb5645f0a4"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578938    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578942    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578945    2576 flags.go:64] FLAG: --register-node="true"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578948    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578955    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578958    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578961    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578964    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 01 19:58:02.584770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578969    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578972    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578975    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578978    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578981    2576 flags.go:64] FLAG: --runonce="false"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578984    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578987    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578990    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578993    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578996    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.578999    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579004    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579007    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579010    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579014    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579017    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579020    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579023    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579026    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579029    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579034    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579037    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579040    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579044    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579047    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 19:58:02.585415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579050    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579053    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579056    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579060    2576 flags.go:64] FLAG: --v="2"
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579065    2576 flags.go:64] FLAG: --version="false"
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579069    2576 flags.go:64] FLAG: --vmodule=""
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579074    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579077    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579223    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579227    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579231    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579235    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579238    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579241    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579243    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579246    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579248    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579251    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579257    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579260    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579262    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.586046 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579265    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579268    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579270    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579273    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579276    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579279    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579281    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579284    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579287    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579289    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579292    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579294    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579297    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579300    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579304    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579306    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579309    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579312    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579315    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.586542 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579317    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579320    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579325    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579327    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579330    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579332    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579335    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579338    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579340    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579344    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579353    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579356    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579358    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579361    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579364    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579367    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579369    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579372    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579374    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.587010 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579377    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579382    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579384    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579387    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579389    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579392    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579396    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579399    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579401    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579404    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579406    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579409    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579411    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579414    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579416    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579419    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579421    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579424    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.587478 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579426    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579429    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579432    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579435    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579438    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579447    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579450    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579452    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579455    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579457    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579460    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579464    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579468    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579471    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579474    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579477    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579480    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579483    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579485    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.587935 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579488    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579491    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579494    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579496    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579499    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579501    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579504    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579506    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579509    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579511    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579514    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579517    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579520    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579522    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579525    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579528    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.588378 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.579557    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579601    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579606    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579610    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579613    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579616    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579618    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579621    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579624    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579626    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579629    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579631    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579634    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579637    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579639    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579642    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579658    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579662    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579665    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579668    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.588801 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579671    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579673    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579676    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579679    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579682    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579684    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579688    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579692    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579694    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579697    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579700    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579703    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579706    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579710    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579712    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579716    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579719    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579721    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579724    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579726    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.589269 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579729    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579732    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579734    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579737    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579739    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579742    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579745    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579747    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579750    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579754    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579756    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579759    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579762    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579764    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579767    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579769    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579772    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579775    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579778    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.589759 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579780    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579783    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579786    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579788    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579791    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579793    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579797    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579800    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579802    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579805    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579808    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579811    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579814    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579816    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579819    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579821    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579824    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579826    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579829    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579831    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.590205 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579834    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579836    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579840    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579843    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579845    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579848    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579850    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579853    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579855    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579858    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579860    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579863    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579865    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579868    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579870    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579873    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579875    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579878    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579882    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.590682 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579884    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579887    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579889    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579892    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579895    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579898    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579901    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579903    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.579908    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.587310    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.587327    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.588101    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.588108    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588172    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588178    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588181    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588185    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588188    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588190    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.591166 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588193    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588196    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588198    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588201    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588204    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588207    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588210    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588212    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588215    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588218    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588220    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588223    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588225    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588228    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588230    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588233    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588235    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588238    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588240    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588243    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.591606 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588246    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588249    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588253    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588258    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588261    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588264    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588266    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588269    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588272    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588274    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588277    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588280    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588283    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588285    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588288    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588291    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588294    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588297    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588299    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.592138 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588302    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588305    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588307    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588310    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588313    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588315    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588318    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588320    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588323    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588325    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588329    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588332    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588335    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588337    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588340    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588343    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588345    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588349    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588351    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588354    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.592586 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588357    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588359    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588362    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588365    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588368    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588370    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588373    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588375    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588378    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588380    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588383    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588386    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588390    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588394    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588397    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588400    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588403    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588406    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588408    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.593150 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588411    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588413    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588416    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588419    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588421    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588424    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588426    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588429    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588432    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588435    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588438    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588441    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588444    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588446    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588449    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588451    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588454    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588456    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588462    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.593687 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588464    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588467    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.588498    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588679    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588685    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588689    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588692    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588695    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588698    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588701    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588704    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588707    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588709    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588712    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588715    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.594148 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588717    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588720    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588724    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588728    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588731    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588734    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588737    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588740    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588742    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588746    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588748    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588751    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588753    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588756    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588758    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588761    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588763    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588766    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588768    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588771    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.594536 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588773    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588776    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588778    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588780    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588783    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588785    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588788    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588790    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588793    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588795    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588798    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588801    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588804    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588807    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588809    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588812    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588815    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588817    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588820    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.595019 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588823    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588825    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588828    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588839    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588842    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588845    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588847    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588850    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588852    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588855    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588857    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588860    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588862    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588865    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588867    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588870    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588872    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588875    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588877    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.595728 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588880    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588882    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588885    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588888    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588892    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588902    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588905    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588908    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588911    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588914    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588916    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588919    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588921    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588924    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588926    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588929    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588931    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588940    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588964    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588968    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.596493 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588971    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588974    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588977    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588979    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588982    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588985    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588987    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588990    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588992    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588995    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.588997    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589000    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589002    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589005    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589007    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589010    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.597250 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.589034    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589071    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589075    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589078    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589081    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589083    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589086    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589089    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589092    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589094    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589097    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589099    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589102    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589104    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589116    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589119    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589122    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589124    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589127    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:02.597640 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589130    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589133    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589135    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589138    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589142    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589145    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589149    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589151    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589154    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589157    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589159    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589162    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589167    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589169    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589172    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589174    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589177    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589180    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589182    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589185    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:02.598403 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589187    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589190    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589192    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589195    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589197    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589199    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589202    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589205    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589213    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589216    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589219    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589221    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589224    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589227    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589230    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589232    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589235    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589237    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589240    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589242    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:02.599144 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589245    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589247    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589250    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589253    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589257    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589259    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589262    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589264    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589267    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589270    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589273    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589276    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589278    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589281    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589284    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589287    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589290    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589292    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589295    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:02.599805 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589298    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589300    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589308    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589311    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589315    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589318    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589321    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589324    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589327    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589329    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589333    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589335    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589338    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589341    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589343    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589346    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589348    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589352    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589354    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:02.600264 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589357    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589360    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589362    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589365    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589367    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589369    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589372    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589374    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589377    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:02.589379    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.589445    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.591886    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.592919    2576 server.go:1014] "Starting client certificate rotation"
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.593074    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.593137    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 19:58:02.600767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.595366    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:58:02.602313 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.602292    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:58:02.610500 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.610477    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 01 19:58:02.614322 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.614305    2576 log.go:25] "Validated CRI v1 image API"
Nov 01 19:58:02.615121 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.615098    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 19:58:02.618054 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.618030    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8c64e93a-ec08-4559-8e06-95751b690061:/dev/nvme0n1p3 9baf065a-c77b-47e2-a340-35fd9130e602:/dev/nvme0n1p4]
Nov 01 19:58:02.618128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.618052    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 19:58:02.623567 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.623442    2576 manager.go:217] Machine: {Timestamp:2026-11-01 19:58:02.621908043 +0000 UTC m=+0.360253087 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102788 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbdf649bb8347d93c5c2823ebb315 SystemUUID:ec2fbdf6-49bb-8347-d93c-5c2823ebb315 BootID:a54a3cd9-b05a-421b-971c-85683fa1a77f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:ee:5d:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:ee:5d:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:47:fb:ca:5a:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 19:58:02.623567 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.623559    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 19:58:02.623773 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.623745    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 19:58:02.627320 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.627290    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 19:58:02.627467 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.627322    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-52.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 19:58:02.627518 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.627469    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 19:58:02.627518 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.627477    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 19:58:02.627518 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.627493    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:58:02.627518 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.627511    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:58:02.628421 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.628401    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 19:58:02.628792 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.628781    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 19:58:02.633206 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633193    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 19:58:02.633332 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633323    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 19:58:02.633373 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633358    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 19:58:02.633373 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633368    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 19:58:02.633483 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633379    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 19:58:02.633483 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633390    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 01 19:58:02.633483 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633399    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 19:58:02.633830 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.633809    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:58:02.636451 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.636431    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 19:58:02.644260 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.644227    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lr2r2"
Nov 01 19:58:02.644610 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.644595    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:58:02.645703 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645686    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 19:58:02.645770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645729    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 19:58:02.645954 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645940    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 19:58:02.646009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645963    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 19:58:02.646009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645974    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 19:58:02.646009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645982    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 19:58:02.646009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.645993    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 19:58:02.646009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646004    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 19:58:02.646872 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646859    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 19:58:02.646872 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646873    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 19:58:02.646933 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646882    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 19:58:02.646933 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646889    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 19:58:02.646933 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646898    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 19:58:02.646933 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.646907    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 19:58:02.648052 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.648042    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 19:58:02.648093 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.648055    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 19:58:02.648851 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.648827    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 19:58:02.648939 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.648852    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 19:58:02.652462 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.652439    2576 server.go:1263] "Started kubelet"
Nov 01 19:58:02.652561 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.652530    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 19:58:02.652609 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.652582    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lr2r2"
Nov 01 19:58:02.652775 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.652734    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 19:58:02.652830 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.652793    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 19:58:02.653281 ip-10-0-133-52 systemd[1]: Started Kubernetes Kubelet.
Nov 01 19:58:02.654692 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.654676    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 19:58:02.655229 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.655211    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 19:58:02.657182 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.657166    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 19:58:02.660146 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.659096    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-52.us-east-2.compute.internal.18e4041ef7c057f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-52.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 19:58:02.652407793 +0000 UTC m=+0.390752839,LastTimestamp:2026-11-01 19:58:02.652407793 +0000 UTC m=+0.390752839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-52.us-east-2.compute.internal,}"
Nov 01 19:58:02.660866 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.660845    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 19:58:02.663451 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.663434    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 19:58:02.663964 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.663947    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 19:58:02.664115 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.664089    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:02.664903 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.664884    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 19:58:02.664903 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.664884    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 19:58:02.665008 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.664916    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 19:58:02.665038 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.665012    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 19:58:02.665038 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.665023    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 19:58:02.665882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.665866    2576 factory.go:55] Registering systemd factory
Nov 01 19:58:02.665935 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.665899    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 01 19:58:02.666159 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.666146    2576 factory.go:153] Registering CRI-O factory
Nov 01 19:58:02.666208 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.666161    2576 factory.go:223] Registration of the crio container factory successfully
Nov 01 19:58:02.666208 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.666202    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 19:58:02.666293 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.666216    2576 factory.go:103] Registering Raw factory
Nov 01 19:58:02.666293 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.666226    2576 manager.go:1199] Started watching for new ooms in manager
Nov 01 19:58:02.667254 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.667240    2576 manager.go:319] Starting recovery of all containers
Nov 01 19:58:02.681015 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.680989    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:02.681811 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.681793    2576 manager.go:324] Recovery completed
Nov 01 19:58:02.683597 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.683579    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-52.us-east-2.compute.internal\" not found" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.685680 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.685665    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.687596 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.687576    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.687702 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.687613    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.687702 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.687629    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.688193 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688180    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 01 19:58:02.688255 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688192    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 19:58:02.688255 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688213    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 19:58:02.688255 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688180    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.688956 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688940    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.689021 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688969    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.689021 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.688979    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.690477 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.690462    2576 policy_none.go:50] "Start"
Nov 01 19:58:02.690545 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.690484    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 19:58:02.690545 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.690497    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 19:58:02.692545 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.692533    2576 policy_none.go:44] "Start"
Nov 01 19:58:02.780208 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.764792    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:02.808979 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.808942    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 19:58:02.810193 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.810173    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 19:58:02.811100 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.811084    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 19:58:02.811157 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.811131    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 19:58:02.811200 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.811181    2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 19:58:02.814953 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.814938    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:02.865405 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.865386    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:02.873213 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.873178    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 19:58:02.875831 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.875816    2576 manager.go:341] "Starting Device Plugin manager"
Nov 01 19:58:02.875935 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.875921    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 19:58:02.876037 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.875936    2576 server.go:87] "Starting device plugin registration server"
Nov 01 19:58:02.876219 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.876206    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 19:58:02.876286 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.876221    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 19:58:02.876350 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.876336    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 19:58:02.876413 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.876403    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 19:58:02.876413 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.876412    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 19:58:02.877093 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.877077    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 19:58:02.877167 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.877136    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:02.911734 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.911698    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"]
Nov 01 19:58:02.911831 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.911773    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.912842 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.912826    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.912919 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.912855    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.912919 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.912867    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.915165 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915153    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.915299 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915284    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.915348 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915316    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.915943 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915930    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.916019 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915946    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.916019 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915957    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.916019 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915971    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.916019 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915974    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.916019 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.915982    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.917969 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.917956    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.918012 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.917988    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.918738 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.918721    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.918821 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.918751    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.918821 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.918762    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.947545 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.947514    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.953769 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.953747    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.967655 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.967611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.967783 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.967668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.967783 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.967724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94566354f58b906840cf934237faff37-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"94566354f58b906840cf934237faff37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.976781 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.976764    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:02.977683 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.977664    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:02.977782 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.977692    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:02.977782 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.977705    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:02.977782 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.977740    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.985694 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:02.985681    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:02.985757 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:02.985703    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-52.us-east-2.compute.internal\": node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.000867 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.000845    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.068783 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.068740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94566354f58b906840cf934237faff37-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"94566354f58b906840cf934237faff37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.068783 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.068788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.068929 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.068814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.068929 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.068873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.068929 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.068880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.068929 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.068873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94566354f58b906840cf934237faff37-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"94566354f58b906840cf934237faff37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.101094 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.101032    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.201738 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.201701    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.250940 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.250911    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.257575 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.257452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.303368 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.303317    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.405348 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.405304    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.507277 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.507242    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.593394 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.593358    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 19:58:03.593924 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.593550    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 19:58:03.593924 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.593577    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 19:58:03.607873 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.607841    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.658618 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.658513    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:53:03 +0000 UTC" deadline="2028-06-20 22:31:29.172665485 +0000 UTC"
Nov 01 19:58:03.658618 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.658560    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14330h33m25.514108259s"
Nov 01 19:58:03.708570 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.708529    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.809445 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:03.809417    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-52.us-east-2.compute.internal\" not found"
Nov 01 19:58:03.812971 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.812949    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:03.865219 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.865179    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.874100 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.874075    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:58:03.875978 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.875958    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:58:03.886225 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.886203    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:58:03.959619 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:03.959568    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c237e4f4c3f7a42c5e7f09a4fd5922.slice/crio-a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141 WatchSource:0}: Error finding container a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141: Status 404 returned error can't find the container with id a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141
Nov 01 19:58:03.963962 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:03.963932    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94566354f58b906840cf934237faff37.slice/crio-f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd WatchSource:0}: Error finding container f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd: Status 404 returned error can't find the container with id f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd
Nov 01 19:58:03.965692 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:03.965675    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 19:58:04.113616 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.113583    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:04.619928 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.619893    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:04.634256 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.634227    2576 apiserver.go:52] "Watching apiserver"
Nov 01 19:58:04.643367 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.642478    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 19:58:04.643367 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.642937    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-kqhqv","openshift-multus/network-metrics-daemon-4jmtc","openshift-dns/node-resolver-cqvzs","openshift-image-registry/node-ca-skvhv","openshift-multus/multus-nd5rg","openshift-ovn-kubernetes/ovnkube-node-rl4sb","kube-system/global-pull-secret-syncer-cmf5w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-4ttgz","openshift-network-operator/iptables-alerter-nl7sv","kube-system/konnectivity-agent-jtnpv","openshift-network-diagnostics/network-check-target-wg7v7"]
Nov 01 19:58:04.645832 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.645692    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.648290 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.648062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.650926 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.650901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:04.651354 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.651305    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:04.653283 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653259    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 19:58:04.653393 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653366    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.653393 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653377    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 19:58:04.653539 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653520    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.653628 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653592    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 19:58:04.653862 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653843    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 19:58:04.653928 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653521    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 19:58:04.653928 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653898    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-754ws\""
Nov 01 19:58:04.654002 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.653975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.654500 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.654482    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 19:58:04.654601 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.654571    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-j6fxj\""
Nov 01 19:58:04.656618 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.655590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.657442 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.657414    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.657548 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.657530    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qrts\""
Nov 01 19:58:04.659991 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.659973    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.660665 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.660628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.663429 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.663157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.664824 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.664803    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 19:58:04.665950 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.665844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:04.666037 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.665962    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:04.668378 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.668355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.671168 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.671141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.671366 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.671329    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:04.673899 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.673876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.676560 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.676540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.679873 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.679849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-daemon-config\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.679962 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.679895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.679962 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.679923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-log-socket\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.679962 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.679948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-device-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.680128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.679975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-script-lib\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.680128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f06e-9873-48b5-8f8a-29b93691922e-konnectivity-ca\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.680128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-k8s-cni-cncf-io\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.680128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-sys-fs\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.680128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nl2c\" (UniqueName: \"kubernetes.io/projected/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kube-api-access-4nl2c\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.680128 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-slash\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-serviceca\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-conf-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-ovn\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-netd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94fk\" (UniqueName: \"kubernetes.io/projected/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-kube-api-access-x94fk\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-cni-binary-copy\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-netns\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-multus-certs\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.680433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxgf\" (UniqueName: \"kubernetes.io/projected/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-kube-api-access-wsxgf\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-bin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-multus\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-hostroot\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovn-node-metrics-cert\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp66t\" (UniqueName: \"kubernetes.io/projected/8bf82705-43f4-4c30-a6f3-403bf0f0d899-kube-api-access-zp66t\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-system-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-systemd-units\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-systemd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-etc-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf82705-43f4-4c30-a6f3-403bf0f0d899-tmp-dir\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-etc-kubernetes\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-node-log\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-bin\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681057 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-config\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdqf\" (UniqueName: \"kubernetes.io/projected/b01e133a-c836-48dd-8c0b-aa317f79d00d-kube-api-access-6bdqf\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.680981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-socket-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxg87\" (UniqueName: \"kubernetes.io/projected/07560b5d-9b60-4ba1-9811-c308393b9ada-kube-api-access-mxg87\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-env-overrides\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8bf82705-43f4-4c30-a6f3-403bf0f0d899-hosts-file\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-cnibin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cf8f06e-9873-48b5-8f8a-29b93691922e-agent-certs\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-host\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-socket-dir-parent\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-netns\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-kubelet\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-kubelet\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-var-lib-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.681925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.681279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-registration-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.697406 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.697368    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 19:58:04.719867 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.719826    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hgxn"
Nov 01 19:58:04.722213 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.722004    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:04.728399 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.728376    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hgxn"
Nov 01 19:58:04.759537 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.759507    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.759702 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.759570    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.759702 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.759620    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.759852 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.759832    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.760308 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.759796    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 19:58:04.760427 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.760406    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 19:58:04.760754 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.760734    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.761100 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761076    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8578\""
Nov 01 19:58:04.761173 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761144    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-24gff\""
Nov 01 19:58:04.761232 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761179    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.761287 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761274    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 19:58:04.761415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761398    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbpdw\""
Nov 01 19:58:04.761608 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761585    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.761784 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761762    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 19:58:04.761949 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.760746    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 19:58:04.762013 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.761985    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 19:58:04.762129 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.762068    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.762386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.762369    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:04.762456 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.762422    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 19:58:04.762674 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.762649    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6s44\""
Nov 01 19:58:04.762845 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.762829    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tdbxs\""
Nov 01 19:58:04.762958 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.762937    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:04.763065 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.763023    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgwx5\""
Nov 01 19:58:04.766593 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.766571    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 19:58:04.782262 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-etc-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf82705-43f4-4c30-a6f3-403bf0f0d899-tmp-dir\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-iptables-alerter-script\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-lib-modules\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-etc-kubernetes\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-node-log\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.782443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-bin\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6854b\" (UniqueName: \"kubernetes.io/projected/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-kube-api-access-6854b\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-system-cni-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-config\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdqf\" (UniqueName: \"kubernetes.io/projected/b01e133a-c836-48dd-8c0b-aa317f79d00d-kube-api-access-6bdqf\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-kubelet-config\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-tmp\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvcr\" (UniqueName: \"kubernetes.io/projected/934b3987-3764-417a-8e86-353093bddc70-kube-api-access-5qvcr\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-socket-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-conf\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prnc\" (UniqueName: \"kubernetes.io/projected/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-kube-api-access-6prnc\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-run\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.782844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxg87\" (UniqueName: \"kubernetes.io/projected/07560b5d-9b60-4ba1-9811-c308393b9ada-kube-api-access-mxg87\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-env-overrides\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8bf82705-43f4-4c30-a6f3-403bf0f0d899-hosts-file\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-dbus\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-systemd\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-host\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-cnibin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cf8f06e-9873-48b5-8f8a-29b93691922e-agent-certs\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.782988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-sys\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-cnibin\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-host\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-socket-dir-parent\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-netns\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-kubelet\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-kubelet\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-var-lib-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.783715 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-registration-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-modprobe-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-daemon-config\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-log-socket\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-device-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-script-lib\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f06e-9873-48b5-8f8a-29b93691922e-konnectivity-ca\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-tuned\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-k8s-cni-cncf-io\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-sys-fs\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nl2c\" (UniqueName: \"kubernetes.io/projected/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kube-api-access-4nl2c\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-host-slash\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-slash\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-serviceca\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-conf-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.784564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-ovn\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-netd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-kubernetes\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-var-lib-kubelet\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94fk\" (UniqueName: \"kubernetes.io/projected/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-kube-api-access-x94fk\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-cni-binary-copy\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-netns\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-host\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.783878    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.783966    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:58:05.283930177 +0000 UTC m=+3.022275233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.783993    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-socket-dir-parent\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-netns\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-kubelet\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784197    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-etc-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.785300 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-kubelet\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-var-lib-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf82705-43f4-4c30-a6f3-403bf0f0d899-tmp-dir\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-registration-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784554    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-etc-kubernetes\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784581    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784669    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-sys-fs\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-slash\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.784957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8bf82705-43f4-4c30-a6f3-403bf0f0d899-hosts-file\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785023    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-cnibin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785227    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f06e-9873-48b5-8f8a-29b93691922e-konnectivity-ca\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-k8s-cni-cncf-io\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-serviceca\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785390    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-env-overrides\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785834    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-log-socket\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-multus-certs\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786080 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.785919    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-config\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-netns\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-netd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786127    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-ovn\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-socket-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786185    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-conf-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-node-log\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-multus-certs\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786430    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-bin\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxgf\" (UniqueName: \"kubernetes.io/projected/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-kube-api-access-wsxgf\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-bin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-daemon-config\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.786998 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-multus\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-device-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-hostroot\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovn-node-metrics-cert\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786767    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-multus\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp66t\" (UniqueName: \"kubernetes.io/projected/8bf82705-43f4-4c30-a6f3-403bf0f0d899-kube-api-access-zp66t\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.786967    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-hostroot\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysconfig\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-system-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-systemd-units\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-systemd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-bin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787203    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-system-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-systemd-units\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.787741 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-systemd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.788373 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.787848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-script-lib\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.788373 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.788054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-cni-binary-copy\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.790459 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.790435    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:04.790459 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.790462    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:04.790629 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.790473    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:04.790629 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.790539    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:58:05.290524708 +0000 UTC m=+3.028869758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:04.792940 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.792876    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cf8f06e-9873-48b5-8f8a-29b93691922e-agent-certs\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.793592 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.793565    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovn-node-metrics-cert\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.794735 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.794690    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nl2c\" (UniqueName: \"kubernetes.io/projected/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kube-api-access-4nl2c\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:04.796284 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.796257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxgf\" (UniqueName: \"kubernetes.io/projected/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-kube-api-access-wsxgf\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:04.797664 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.797493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdqf\" (UniqueName: \"kubernetes.io/projected/b01e133a-c836-48dd-8c0b-aa317f79d00d-kube-api-access-6bdqf\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.798085 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.798057    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp66t\" (UniqueName: \"kubernetes.io/projected/8bf82705-43f4-4c30-a6f3-403bf0f0d899-kube-api-access-zp66t\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.798182 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.798122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxg87\" (UniqueName: \"kubernetes.io/projected/07560b5d-9b60-4ba1-9811-c308393b9ada-kube-api-access-mxg87\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:04.799923 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.799902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94fk\" (UniqueName: \"kubernetes.io/projected/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-kube-api-access-x94fk\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:04.818009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.817963    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"94566354f58b906840cf934237faff37","Type":"ContainerStarted","Data":"f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd"}
Nov 01 19:58:04.819317 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.819291    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"d5c237e4f4c3f7a42c5e7f09a4fd5922","Type":"ContainerStarted","Data":"a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141"}
Nov 01 19:58:04.887597 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-lib-modules\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887597 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6854b\" (UniqueName: \"kubernetes.io/projected/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-kube-api-access-6854b\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.887597 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-system-cni-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-kubelet-config\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-tmp\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvcr\" (UniqueName: \"kubernetes.io/projected/934b3987-3764-417a-8e86-353093bddc70-kube-api-access-5qvcr\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-conf\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prnc\" (UniqueName: \"kubernetes.io/projected/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-kube-api-access-6prnc\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-run\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-dbus\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-lib-modules\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-systemd\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-host\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-sys\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-cnibin\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-modprobe-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.887896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-system-cni-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-tuned\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-host-slash\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.887980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-kubernetes\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-var-lib-kubelet\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-conf\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-kubelet-config\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysconfig\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.888162    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-iptables-alerter-script\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:04.888228    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:05.388195356 +0000 UTC m=+3.126540407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-run\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.888621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-dbus\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-host-slash\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888527    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-kubernetes\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888578    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-var-lib-kubelet\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888718    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888899    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-iptables-alerter-script\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888953    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888977    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-systemd\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.888996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-host\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.889008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysconfig\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.889038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-sys\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.889069    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-cnibin\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.889089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-modprobe-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.889415 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.889250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.890845 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.890818    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-tmp\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.891616 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.891589    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-tuned\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.897368 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.897305    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prnc\" (UniqueName: \"kubernetes.io/projected/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-kube-api-access-6prnc\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:04.897368 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.897305    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvcr\" (UniqueName: \"kubernetes.io/projected/934b3987-3764-417a-8e86-353093bddc70-kube-api-access-5qvcr\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:04.897368 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.897343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6854b\" (UniqueName: \"kubernetes.io/projected/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-kube-api-access-6854b\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:04.986026 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.985982    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:04.992139 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.991899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:04.997260 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.997236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:58:04.998411 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:04.998382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:58:05.000049 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:04.999988    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf8f06e_9873_48b5_8f8a_29b93691922e.slice/crio-98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174 WatchSource:0}: Error finding container 98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174: Status 404 returned error can't find the container with id 98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174
Nov 01 19:58:05.006886 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.006858    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01e133a_c836_48dd_8c0b_aa317f79d00d.slice/crio-8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2 WatchSource:0}: Error finding container 8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2: Status 404 returned error can't find the container with id 8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2
Nov 01 19:58:05.011233 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.010592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nd5rg"
Nov 01 19:58:05.011233 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.011046    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf82705_43f4_4c30_a6f3_403bf0f0d899.slice/crio-2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974 WatchSource:0}: Error finding container 2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974: Status 404 returned error can't find the container with id 2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974
Nov 01 19:58:05.017482 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.017457    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae3e006_540b_4bd3_b35e_7c3a9e1d130f.slice/crio-60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4 WatchSource:0}: Error finding container 60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4: Status 404 returned error can't find the container with id 60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4
Nov 01 19:58:05.023694 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.023654    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07560b5d_9b60_4ba1_9811_c308393b9ada.slice/crio-8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0 WatchSource:0}: Error finding container 8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0: Status 404 returned error can't find the container with id 8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0
Nov 01 19:58:05.024839 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.024816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:58:05.027573 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.027551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:58:05.037564 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.037541    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f03a9b_91cc_4cd0_b6a4_ad246a1ac090.slice/crio-b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0 WatchSource:0}: Error finding container b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0: Status 404 returned error can't find the container with id b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0
Nov 01 19:58:05.041047 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.041024    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded76efa0_e418_4a93_8b3f_5d95d7e5f33f.slice/crio-8672e083e85ea56c754eabe76e1cb92cd2d5752f87511013481b51064f45bc87 WatchSource:0}: Error finding container 8672e083e85ea56c754eabe76e1cb92cd2d5752f87511013481b51064f45bc87: Status 404 returned error can't find the container with id 8672e083e85ea56c754eabe76e1cb92cd2d5752f87511013481b51064f45bc87
Nov 01 19:58:05.056986 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.056964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:58:05.059159 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.059133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:58:05.071601 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.071564    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4efd09_675b_47ab_b4cc_c4da33b77d6a.slice/crio-1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc WatchSource:0}: Error finding container 1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc: Status 404 returned error can't find the container with id 1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc
Nov 01 19:58:05.074082 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:05.074057    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934b3987_3764_417a_8e86_353093bddc70.slice/crio-989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505 WatchSource:0}: Error finding container 989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505: Status 404 returned error can't find the container with id 989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505
Nov 01 19:58:05.291375 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.291324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:05.291546 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.291393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:05.291546 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.291522    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:05.291708 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.291588    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:58:06.291568763 +0000 UTC m=+4.029913813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:05.292060 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.292039    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:05.292157 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.292066    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:05.292157 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.292077    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:05.292157 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.292116    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:58:06.292104477 +0000 UTC m=+4.030449522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:05.391837 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.391753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:05.392004 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.391950    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:05.392062 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.392034    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:06.392009867 +0000 UTC m=+4.130354900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:05.732355 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.732282    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:53:05 +0000 UTC" deadline="2028-06-22 08:16:05.899586935 +0000 UTC"
Nov 01 19:58:05.732355 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.732350    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14364h18m0.167241837s"
Nov 01 19:58:05.811693 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.811612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:05.811986 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.811952    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:05.812465 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.812443    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:05.812597 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.812567    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:05.812663 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.812639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:05.812807 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:05.812785    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:05.840211 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.840174    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cqvzs" event={"ID":"8bf82705-43f4-4c30-a6f3-403bf0f0d899","Type":"ContainerStarted","Data":"2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974"}
Nov 01 19:58:05.842878 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.842845    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2"}
Nov 01 19:58:05.854242 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.854211    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jtnpv" event={"ID":"0cf8f06e-9873-48b5-8f8a-29b93691922e","Type":"ContainerStarted","Data":"98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174"}
Nov 01 19:58:05.859046 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.859015    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerStarted","Data":"989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505"}
Nov 01 19:58:05.862767 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.862699    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kqhqv" event={"ID":"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f","Type":"ContainerStarted","Data":"8672e083e85ea56c754eabe76e1cb92cd2d5752f87511013481b51064f45bc87"}
Nov 01 19:58:05.866688 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.866606    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p" event={"ID":"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090","Type":"ContainerStarted","Data":"b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0"}
Nov 01 19:58:05.869070 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.868991    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nl7sv" event={"ID":"0a4efd09-675b-47ab-b4cc-c4da33b77d6a","Type":"ContainerStarted","Data":"1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc"}
Nov 01 19:58:05.884624 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.884582    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nd5rg" event={"ID":"07560b5d-9b60-4ba1-9811-c308393b9ada","Type":"ContainerStarted","Data":"8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0"}
Nov 01 19:58:05.896946 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:05.896909    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-skvhv" event={"ID":"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f","Type":"ContainerStarted","Data":"60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4"}
Nov 01 19:58:06.305496 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.305455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:06.305646 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.305538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:06.305755 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.305736    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:06.305841 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.305813    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:58:08.305793129 +0000 UTC m=+6.044138162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:06.306268 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.306250    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:06.306365 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.306272    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:06.306365 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.306294    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:06.306365 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.306347    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:58:08.306328376 +0000 UTC m=+6.044673408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:06.406820 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.406774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:06.407027 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.407006    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:06.407095 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:06.407084    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:08.407064088 +0000 UTC m=+6.145409142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:06.733230 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.732933    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:53:05 +0000 UTC" deadline="2028-05-07 08:01:19.692377573 +0000 UTC"
Nov 01 19:58:06.733230 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.733232    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13260h3m12.959153967s"
Nov 01 19:58:06.915824 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.915465    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5c237e4f4c3f7a42c5e7f09a4fd5922" containerID="fe87fe6355985d48991bc6fea94cf588e5e68f78edff9ef839fb64b22a4362ac" exitCode=0
Nov 01 19:58:06.916150 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.916038    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"d5c237e4f4c3f7a42c5e7f09a4fd5922","Type":"ContainerDied","Data":"fe87fe6355985d48991bc6fea94cf588e5e68f78edff9ef839fb64b22a4362ac"}
Nov 01 19:58:06.936479 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.935104    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"94566354f58b906840cf934237faff37","Type":"ContainerStarted","Data":"077f4e8b23014b2bf95d74c9ed6b77e6e27ecbb7f2baaa33037291a59c4b2db0"}
Nov 01 19:58:06.951404 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:06.951311    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal" podStartSLOduration=2.9512906450000003 podStartE2EDuration="2.951290645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:06.950993586 +0000 UTC m=+4.689338642" watchObservedRunningTime="2026-11-01 19:58:06.951290645 +0000 UTC m=+4.689635700"
Nov 01 19:58:07.812311 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:07.812224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:07.812955 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:07.812369    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:07.812955 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:07.812815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:07.812955 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:07.812915    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:07.813173 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:07.812984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:07.813173 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:07.813066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:07.965054 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:07.964958    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"d5c237e4f4c3f7a42c5e7f09a4fd5922","Type":"ContainerStarted","Data":"16237f23437c9ff3dfb89ef33698230823f6b7f4cea5f986d47299e2c7c4d18d"}
Nov 01 19:58:07.981130 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:07.979849    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" podStartSLOduration=3.979829441 podStartE2EDuration="3.979829441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:07.979652733 +0000 UTC m=+5.717997786" watchObservedRunningTime="2026-11-01 19:58:07.979829441 +0000 UTC m=+5.718174497"
Nov 01 19:58:08.327352 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:08.327316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:08.327530 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:08.327430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:08.327642 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.327615    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:08.327642 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.327634    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:08.327781 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.327650    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:08.327781 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.327721    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:58:12.327701686 +0000 UTC m=+10.066046737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:08.327781 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.327765    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:08.327969 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.327809    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:58:12.327795241 +0000 UTC m=+10.066140287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:08.428383 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:08.428347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:08.428547 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.428527    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:08.428706 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:08.428609    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:12.428590001 +0000 UTC m=+10.166935034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:08.976997 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:08.976234    2576 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="0e997eee7b852bddc0e6f001fd3cbe2f3bb8c190f1a59f78ab5c5f50c627fc40" exitCode=0
Nov 01 19:58:08.976997 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:08.976904    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerDied","Data":"0e997eee7b852bddc0e6f001fd3cbe2f3bb8c190f1a59f78ab5c5f50c627fc40"}
Nov 01 19:58:09.812534 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:09.811631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:09.812534 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:09.811791    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:09.812534 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:09.812225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:09.812534 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:09.812330    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:09.812534 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:09.812397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:09.812534 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:09.812467    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:09.981224 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:09.981173    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jtnpv" event={"ID":"0cf8f06e-9873-48b5-8f8a-29b93691922e","Type":"ContainerStarted","Data":"b058230a3f974e0598591a3f5ba1a2e3338d40d77511c2a2d08b2d2290c51ae0"}
Nov 01 19:58:10.989497 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:10.989451    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-skvhv" event={"ID":"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f","Type":"ContainerStarted","Data":"3edb77a6f8c093c92867e219d97427a3c1251fb48deecb53d982f2e45e41489f"}
Nov 01 19:58:11.004881 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:11.004140    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jtnpv" podStartSLOduration=3.758641717 podStartE2EDuration="8.004119401s" totalImagesPullingTime="4.245477684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:09.999045851 +0000 UTC m=+7.737390908" watchObservedRunningTime="2026-11-01 19:58:11.004119401 +0000 UTC m=+8.742464448"
Nov 01 19:58:11.004881 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:11.004546    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-skvhv" podStartSLOduration=2.768130222 podStartE2EDuration="8.0045373s" totalImagesPullingTime="5.236407078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:11.003330567 +0000 UTC m=+8.741675623" watchObservedRunningTime="2026-11-01 19:58:11.0045373 +0000 UTC m=+8.742882356"
Nov 01 19:58:11.811832 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:11.811711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:11.812016 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:11.811863    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:11.812198 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:11.812175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:11.812298 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:11.812240    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:11.812384 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:11.812360    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:11.812471 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:11.812450    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:11.997180 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:11.996873    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p" event={"ID":"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090","Type":"ContainerStarted","Data":"37d447e813ddc8c551423507e447ce0d6cc9dcc439aceac5137edf95998d18f7"}
Nov 01 19:58:12.363399 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:12.363350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:12.363582 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:12.363439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:12.363659 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.363580    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:12.363659 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.363642    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.363624685 +0000 UTC m=+18.101969717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:12.364580 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.364118    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:12.364580 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.364145    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:12.364580 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.364159    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:12.364580 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.364209    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.364193106 +0000 UTC m=+18.102538143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:12.464618 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:12.464106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:12.464618 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.464263    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:12.464618 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:12.464316    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.464303706 +0000 UTC m=+18.202648737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:13.095052 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.095011    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:13.096442 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.096408    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:13.196311 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.196286    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 19:58:13.812919 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.812324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:13.812919 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:13.812457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:13.812919 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.812522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:13.812919 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:13.812587    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:13.812919 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.812676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:13.812919 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:13.812747    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:13.886088 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.885771    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T19:58:13.196309336Z","UUID":"670c00dd-8048-43dc-9cd8-539850396353","Handler":null,"Name":"","Endpoint":""}
Nov 01 19:58:13.887882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.887230    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 19:58:13.887882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:13.887417    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 19:58:14.004149 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:14.004112    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerStarted","Data":"e955a95859bf09adb559ff3f53987938445a9e652abeb383129af1ce0f95c598"}
Nov 01 19:58:14.006196 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:14.006165    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p" event={"ID":"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090","Type":"ContainerStarted","Data":"a15889bf30b661bc1dffb3d3b6c33c4ee5b10d837a3646d86a81831076fb9f9a"}
Nov 01 19:58:14.014155 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:14.014127    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nd5rg" event={"ID":"07560b5d-9b60-4ba1-9811-c308393b9ada","Type":"ContainerStarted","Data":"0289a679808cca9ee496a66037d413ea3b6d8c8578b24f31beee9c070c607732"}
Nov 01 19:58:14.015780 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:14.015756    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cqvzs" event={"ID":"8bf82705-43f4-4c30-a6f3-403bf0f0d899","Type":"ContainerStarted","Data":"e99590e078cfd449f48f1c779597ebccd06b905957266fc541d30f724094bde0"}
Nov 01 19:58:14.039577 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:14.038793    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cqvzs" podStartSLOduration=2.412915839 podStartE2EDuration="11.038775211s" totalImagesPullingTime="8.625859372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:14.038718224 +0000 UTC m=+11.777063276" watchObservedRunningTime="2026-11-01 19:58:14.038775211 +0000 UTC m=+11.777120263"
Nov 01 19:58:14.057022 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:14.056967    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nd5rg" podStartSLOduration=2.716698042 podStartE2EDuration="11.056946015s" totalImagesPullingTime="8.340247973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:14.056353319 +0000 UTC m=+11.794698369" watchObservedRunningTime="2026-11-01 19:58:14.056946015 +0000 UTC m=+11.795291069"
Nov 01 19:58:15.021010 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:15.020968    2576 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="e955a95859bf09adb559ff3f53987938445a9e652abeb383129af1ce0f95c598" exitCode=0
Nov 01 19:58:15.021539 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:15.021062    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerDied","Data":"e955a95859bf09adb559ff3f53987938445a9e652abeb383129af1ce0f95c598"}
Nov 01 19:58:15.025868 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:15.025694    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p" event={"ID":"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090","Type":"ContainerStarted","Data":"8d38d74233f4407b55f3225fd3c51f1e388c6a0355d01f36fb21967c696eb4f5"}
Nov 01 19:58:15.812184 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:15.812103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:15.812184 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:15.812151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:15.812379 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:15.812258    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:15.812635 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:15.812613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:15.812745 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:15.812686    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:15.812798 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:15.812755    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:16.032988 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:16.032882    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kqhqv" event={"ID":"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f","Type":"ContainerStarted","Data":"ae43b92687bdf08433b2e40a8eda384567cc7fc4cda571538e3fc325a4d6b415"}
Nov 01 19:58:16.034471 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:16.034433    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nl7sv" event={"ID":"0a4efd09-675b-47ab-b4cc-c4da33b77d6a","Type":"ContainerStarted","Data":"f292c3b04b5a2143389ff364ab97d9a44ce1ffef41d60547aa0abb684769e0d5"}
Nov 01 19:58:16.048492 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:16.048430    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p" podStartSLOduration=4.123010745 podStartE2EDuration="13.048412518s" totalImagesPullingTime="8.925401773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:15.057706624 +0000 UTC m=+12.796051677" watchObservedRunningTime="2026-11-01 19:58:16.048412518 +0000 UTC m=+13.786757572"
Nov 01 19:58:16.048766 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:16.048737    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nl7sv" podStartSLOduration=4.3458732 podStartE2EDuration="13.048729066s" totalImagesPullingTime="8.702855866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:16.047907608 +0000 UTC m=+13.786252660" watchObservedRunningTime="2026-11-01 19:58:16.048729066 +0000 UTC m=+13.787074118"
Nov 01 19:58:17.038418 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:17.038382    2576 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="4b6fd2cdbb5c58a9deae8eaf10f74eb08f89f326fcc179cce2d1c5bbfe6088ce" exitCode=0
Nov 01 19:58:17.039030 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:17.038487    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerDied","Data":"4b6fd2cdbb5c58a9deae8eaf10f74eb08f89f326fcc179cce2d1c5bbfe6088ce"}
Nov 01 19:58:17.054331 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:17.054271    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kqhqv" podStartSLOduration=3.176002611 podStartE2EDuration="14.05425281s" totalImagesPullingTime="10.878250199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:17.054168934 +0000 UTC m=+14.792513986" watchObservedRunningTime="2026-11-01 19:58:17.05425281 +0000 UTC m=+14.792597861"
Nov 01 19:58:17.812257 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:17.812226    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:17.812386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:17.812352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:17.812386 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:17.812362    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:17.812476 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:17.812385    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:17.812476 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:17.812464    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:17.812592 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:17.812566    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:18.043059 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:18.043023    2576 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="0ebba3b1bfc2c0d6555f8560dcaaf7edafd49077cce69e640d260fccdad72116" exitCode=0
Nov 01 19:58:18.043769 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:18.043079    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerDied","Data":"0ebba3b1bfc2c0d6555f8560dcaaf7edafd49077cce69e640d260fccdad72116"}
Nov 01 19:58:19.811815 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:19.811599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:19.812224 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:19.811682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:19.812224 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:19.811891    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:19.812224 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:19.811692    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:19.812224 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:19.811986    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:19.812224 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:19.812018    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:20.258975 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:20.258942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:20.394977 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:20.394950    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:20.428726 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:20.428636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:20.428811 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:20.428749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:20.428811 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.428755    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:20.428811 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.428775    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:20.428811 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.428785    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:20.428965 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.428834    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:58:36.428814237 +0000 UTC m=+34.167159275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:20.428965 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.428849    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:20.428965 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.428889    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:58:36.428874579 +0000 UTC m=+34.167219624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:20.529285 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:20.529261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:20.529396 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.529375    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:20.529472 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:20.529456    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:36.529434556 +0000 UTC m=+34.267779593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:21.052269 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.052096    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"4c1f11a5dea51a091f56744e922031d2b662d808cc1a245cca98244bb597e8fd"}
Nov 01 19:58:21.052728 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.052306    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"7b4b78fcff66ae9ad4f52e2f100fb739cafb8f20e41250b3810918a69c48ed15"}
Nov 01 19:58:21.052728 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.052336    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"074008ac06622b8582284e6f35e1479df1bc7f7554f217d873307b8f420c05e1"}
Nov 01 19:58:21.052728 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.052366    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"98419bb303af968b8459396473b0c40a00aede1160583f572fc6ee5da8c1bab9"}
Nov 01 19:58:21.052728 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.052392    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"193390aaf5e1b2490977f5313d75153ffa40a49733e5e77c11162b4d801ff549"}
Nov 01 19:58:21.052728 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.052418    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"3a7707c7dbf6b029415dba5965e9c9a48ee337c10ba62e889442aa1d4da99458"}
Nov 01 19:58:21.054891 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.054870    2576 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="42f9678475b0e5e5768ed24c4c961872b073968853ad0f06efc3ba24607025e1" exitCode=0
Nov 01 19:58:21.054989 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.054911    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerDied","Data":"42f9678475b0e5e5768ed24c4c961872b073968853ad0f06efc3ba24607025e1"}
Nov 01 19:58:21.812120 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.812092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:21.812291 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.812123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:21.812291 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:21.812193    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:21.812291 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.812223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:21.812402 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:21.812306    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:21.812402 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:21.812388    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:21.953056 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:21.953028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:22.024429 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:22.024398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:22.060271 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:22.060242    2576 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="82ec516f9e6e7f864575305c4a6c3158d50e490f97e69ef66e79c6cbf284f894" exitCode=0
Nov 01 19:58:22.060601 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:22.060285    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerDied","Data":"82ec516f9e6e7f864575305c4a6c3158d50e490f97e69ef66e79c6cbf284f894"}
Nov 01 19:58:23.066269 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:23.066235    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" event={"ID":"934b3987-3764-417a-8e86-353093bddc70","Type":"ContainerStarted","Data":"223e715467c5c8c94e80207ad3c2f959e01a4329bd3728e5dec836dd5587cc10"}
Nov 01 19:58:23.070756 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:23.070728    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"c931aef5051b7eebe2c11e3433df24a4fdda9de7659b89e0f62931820cafa16d"}
Nov 01 19:58:23.089657 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:23.089606    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4ttgz" podStartSLOduration=8.381125527 podStartE2EDuration="20.089595209s" totalImagesPullingTime="11.708469682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:23.087963181 +0000 UTC m=+20.826308232" watchObservedRunningTime="2026-11-01 19:58:23.089595209 +0000 UTC m=+20.827940281"
Nov 01 19:58:23.812065 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:23.812026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:23.812208 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:23.812099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:23.812208 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:23.812189    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:23.812208 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:23.812196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:23.812365 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:23.812264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:23.812365 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:23.812327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:24.037188 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:24.037157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:25.077535 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.077452    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" event={"ID":"b01e133a-c836-48dd-8c0b-aa317f79d00d","Type":"ContainerStarted","Data":"021601180f9819d7113a506ca1c8cc84ff4219f0e4246f4add7f085808d08ab3"}
Nov 01 19:58:25.078249 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.077698    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:25.092784 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.092760    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:25.103026 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.102988    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb" podStartSLOduration=6.811704829 podStartE2EDuration="22.102976174s" totalImagesPullingTime="15.291271345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:25.101294412 +0000 UTC m=+22.839639463" watchObservedRunningTime="2026-11-01 19:58:25.102976174 +0000 UTC m=+22.841321224"
Nov 01 19:58:25.812465 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.812264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:25.812617 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.812328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:25.812617 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:25.812558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:25.812617 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.812337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:25.812617 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:25.812606    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:25.812798 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:25.812706    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:25.836304 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.836282    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:25.836403 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.836391    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:58:25.836849 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.836832    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:58:25.909481 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:25.909450    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:26.080060 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.079982    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:26.080060 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.080051    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:26.095213 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.095182    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:26.439688 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.439572    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmf5w"]
Nov 01 19:58:26.439855 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.439695    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:26.439855 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:26.439805    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:26.443495 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.443229    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4jmtc"]
Nov 01 19:58:26.443495 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.443341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:26.443495 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:26.443456    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:26.443908 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.443879    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wg7v7"]
Nov 01 19:58:26.443992 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.443963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:26.444083 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:26.444051    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:26.454768 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.454741    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:26.529844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.529812    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:26.619347 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.619302    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:26.721453 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.721416    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:26.821781 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:26.821747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:27.145906 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:27.145831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:27.250562 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:27.250533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:27.345614 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:27.345585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:27.811764 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:27.811726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:27.811946 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:27.811824    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:27.811946 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:27.811881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:27.812044 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:27.811987    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:28.269710 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:28.269672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:28.370204 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:28.370169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:28.609273 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:28.609192    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:28.811527 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:28.811502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:28.811670 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:28.811632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:29.104216 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.104177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:29.228320 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.228290    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqvzs_8bf82705-43f4-4c30-a6f3-403bf0f0d899/dns-node-resolver/0.log"
Nov 01 19:58:29.242467 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.242433    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:29.343775 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.343744    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:29.444975 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.444941    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:29.616571 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.616536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:29.811750 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.811673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:29.811750 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:29.811711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:29.811923 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:29.811785    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:29.811958 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:29.811920    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:30.427709 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:30.427675    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skvhv_4ae3e006-540b-4bd3-b35e-7c3a9e1d130f/node-ca/0.log"
Nov 01 19:58:30.610472 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:30.610440    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:30.811388 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:30.811356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:30.811537 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:30.811478    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:31.024170 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.024090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:31.209237 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.209205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:31.295728 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.295631    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:31.414199 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.414168    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:31.582810 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.582734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:31.811921 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.811880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:31.812120 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.811880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:31.812120 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:31.812015    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:31.812120 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:31.812071    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:31.814536 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:31.814511    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:32.605746 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:32.605712    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:32.812958 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:32.812927    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:32.813114 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:32.813015    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:32.891571 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:32.891488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:33.008373 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.008340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:33.209848 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.209813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:33.310455 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.310421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:33.412889 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.412853    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:33.811490 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.811458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:33.811966 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:33.811563    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:33.811966 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.811622    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:33.811966 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:33.811753    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:33.812288 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.812269    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:33.913347 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:33.913312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.362045 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.362009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.462249 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.462216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.547427 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.547397    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.634068 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.633993    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.701750 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.701716    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.776305 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.776276    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:34.812421 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.812392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:34.812810 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:34.812534    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:34.876754 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:34.876708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.004944 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.004911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.105320 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.105292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.206540 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.206509    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.418219 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.418145    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.602245 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.602209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.805667 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.805618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:35.811578 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.811562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:35.811633 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:35.811590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:35.811697 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:35.811680    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:35.811738 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:35.811719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:36.450157 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.450126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.450172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.450271    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.450290    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.450310    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.450319    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs podName:a3a98578-24d3-4c0b-ba53-d2285c9ed6bd nodeName:}" failed. No retries permitted until 2026-11-01 19:59:08.450307526 +0000 UTC m=+66.188652554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs") pod "network-metrics-daemon-4jmtc" (UID: "a3a98578-24d3-4c0b-ba53-d2285c9ed6bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.450320    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2jnzh for pod openshift-network-diagnostics/network-check-target-wg7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:36.450527 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.450363    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh podName:71e7cc09-97e2-4003-8b99-ee00d3d8252a nodeName:}" failed. No retries permitted until 2026-11-01 19:59:08.450351419 +0000 UTC m=+66.188696461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jnzh" (UniqueName: "kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh") pod "network-check-target-wg7v7" (UID: "71e7cc09-97e2-4003-8b99-ee00d3d8252a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:36.484441 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.484415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:36.551433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.551402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:36.551584 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.551503    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:36.551584 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.551554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret podName:8e0c52c4-326c-4a61-ae5f-b589196d41e6 nodeName:}" failed. No retries permitted until 2026-11-01 19:59:08.551540785 +0000 UTC m=+66.289885814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret") pod "global-pull-secret-syncer-cmf5w" (UID: "8e0c52c4-326c-4a61-ae5f-b589196d41e6") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:36.609175 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.609140    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:36.811873 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.811844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:36.812009 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:36.811986    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:36.877629 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:36.877595    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:37.004158 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.004123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:37.227802 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.227766    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:37.406479 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.406436    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:37.811346 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.811298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:37.811792 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.811352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:37.811792 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:37.811417    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:37.811792 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:37.811523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:37.812335 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.812312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:37.910782 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:37.910747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:38.012188 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:38.012155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:38.606498 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:38.606462    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:38.805293 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:38.805259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:38.812307 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:38.812282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:38.812725 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:38.812431    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:39.206386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:39.206351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:39.409563 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:39.409520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:39.802451 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:39.802421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:39.811768 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:39.811744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:39.811853 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:39.811775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:39.811892 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:39.811867    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:39.811995 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:39.811976    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:40.208807 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:40.208773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:40.404920 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:40.404887    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:40.811543 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:40.811513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:40.811726 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:40.811628    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:41.775404 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:41.775369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:41.811687 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:41.811662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:41.811799 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:41.811663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:41.811799 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:41.811757    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:41.811882 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:41.811824    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:41.848803 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:41.848772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:42.812093 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:42.812058    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:42.812483 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:42.812154    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:43.811995 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:43.811953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:43.812177 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:43.811953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:43.812177 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:43.812082    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:43.812177 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:43.812130    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:44.811877 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:44.811850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:44.812010 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:44.811952    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:45.811702 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:45.811491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:45.812169 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:45.811491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:45.812169 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:45.811799    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:45.812169 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:45.811835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:46.812219 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:46.812187    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:46.812596 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:46.812334    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:47.811528 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:47.811499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:47.811705 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:47.811499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:47.811705 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:47.811596    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:47.811705 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:47.811684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:48.811933 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:48.811900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:48.812372 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:48.812051    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4jmtc" podUID="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd"
Nov 01 19:58:48.895720 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:48.895682    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:49.502237 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:49.502204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:49.596912 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:49.596880    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:49.812117 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:49.812092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:49.812439 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:49.812092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:49.812439 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:49.812198    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmf5w" podUID="8e0c52c4-326c-4a61-ae5f-b589196d41e6"
Nov 01 19:58:49.812439 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:49.812264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wg7v7" podUID="71e7cc09-97e2-4003-8b99-ee00d3d8252a"
Nov 01 19:58:50.025140 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.025066    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeReady"
Nov 01 19:58:50.025277 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.025196    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 19:58:50.067871 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.067833    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dr9z8"]
Nov 01 19:58:50.069402 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.069387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.074332 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.074301    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jv26r"]
Nov 01 19:58:50.075817 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.075798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.078152 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.078131    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 19:58:50.078243 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.078178    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 19:58:50.078243 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.078212    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9pms2\""
Nov 01 19:58:50.078243 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.078231    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 19:58:50.087493 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.087472    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dr9z8"]
Nov 01 19:58:50.091078 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.091054    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jv26r"]
Nov 01 19:58:50.093534 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.093514    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qczmj"]
Nov 01 19:58:50.095433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.095417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.114955 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.114931    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qczmj"]
Nov 01 19:58:50.176340 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176312    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 19:58:50.176572 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176555    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 19:58:50.176635 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176554    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 19:58:50.176635 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176617    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 19:58:50.176798 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176784    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 19:58:50.176848 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176803    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 19:58:50.176848 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176804    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgkqn\""
Nov 01 19:58:50.176919 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.176873    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9xfd7\""
Nov 01 19:58:50.240985 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.240957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62v8\" (UniqueName: \"kubernetes.io/projected/baa37f05-fc26-4a6c-a194-e3b4a88a5941-kube-api-access-m62v8\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.241077 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.240991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8dp\" (UniqueName: \"kubernetes.io/projected/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-api-access-bp8dp\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.241136 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m7v6\" (UniqueName: \"kubernetes.io/projected/34e81519-4820-4e98-976b-48933a0eb319-kube-api-access-5m7v6\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.241172 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-data-volume\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.241172 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa37f05-fc26-4a6c-a194-e3b4a88a5941-tmp-dir\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.241253 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.241285 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/baa37f05-fc26-4a6c-a194-e3b4a88a5941-config-volume\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.241316 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e81519-4820-4e98-976b-48933a0eb319-cert\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.241316 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.241405 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/baa37f05-fc26-4a6c-a194-e3b4a88a5941-metrics-tls\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.241449 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.241406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-crio-socket\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.341917 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.341842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.341917 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.341874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/baa37f05-fc26-4a6c-a194-e3b4a88a5941-config-volume\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.341917 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.341893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e81519-4820-4e98-976b-48933a0eb319-cert\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.342110 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.342110 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/baa37f05-fc26-4a6c-a194-e3b4a88a5941-metrics-tls\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.342110 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-crio-socket\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.342249 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62v8\" (UniqueName: \"kubernetes.io/projected/baa37f05-fc26-4a6c-a194-e3b4a88a5941-kube-api-access-m62v8\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.342249 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8dp\" (UniqueName: \"kubernetes.io/projected/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-api-access-bp8dp\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.342249 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m7v6\" (UniqueName: \"kubernetes.io/projected/34e81519-4820-4e98-976b-48933a0eb319-kube-api-access-5m7v6\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.342395 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-data-volume\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.342395 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa37f05-fc26-4a6c-a194-e3b4a88a5941-tmp-dir\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.342570 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342544    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa37f05-fc26-4a6c-a194-e3b4a88a5941-tmp-dir\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.342570 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.342720 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342583    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/baa37f05-fc26-4a6c-a194-e3b4a88a5941-config-volume\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.342806 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342790    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-data-volume\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.342904 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.342885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-crio-socket\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.345849 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.345825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/baa37f05-fc26-4a6c-a194-e3b4a88a5941-metrics-tls\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.345950 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.345862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.346099 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.346079    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e81519-4820-4e98-976b-48933a0eb319-cert\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.350043 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.350019    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62v8\" (UniqueName: \"kubernetes.io/projected/baa37f05-fc26-4a6c-a194-e3b4a88a5941-kube-api-access-m62v8\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.350333 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.350313    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8dp\" (UniqueName: \"kubernetes.io/projected/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-api-access-bp8dp\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.350592 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.350572    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m7v6\" (UniqueName: \"kubernetes.io/projected/34e81519-4820-4e98-976b-48933a0eb319-kube-api-access-5m7v6\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.380927 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.380902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:58:50.386594 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.386576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:50.409845 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.409820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:58:50.671321 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.671244    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dr9z8"]
Nov 01 19:58:50.674916 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.674893    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jv26r"]
Nov 01 19:58:50.675002 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:50.674913    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e81519_4820_4e98_976b_48933a0eb319.slice/crio-4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902 WatchSource:0}: Error finding container 4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902: Status 404 returned error can't find the container with id 4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902
Nov 01 19:58:50.676860 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:50.676839    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa37f05_fc26_4a6c_a194_e3b4a88a5941.slice/crio-02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2 WatchSource:0}: Error finding container 02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2: Status 404 returned error can't find the container with id 02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2
Nov 01 19:58:50.686704 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.686679    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qczmj"]
Nov 01 19:58:50.689608 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:50.689589    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1f5e20_e8b6_45e3_9d44_1b2f73c3ac3e.slice/crio-de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577 WatchSource:0}: Error finding container de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577: Status 404 returned error can't find the container with id de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577
Nov 01 19:58:50.811939 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.811907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:58:50.815914 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.815868    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\""
Nov 01 19:58:50.816220 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:50.815923    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 19:58:51.139097 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.139063    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dr9z8" event={"ID":"34e81519-4820-4e98-976b-48933a0eb319","Type":"ContainerStarted","Data":"4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902"}
Nov 01 19:58:51.140781 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.140751    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qczmj" event={"ID":"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e","Type":"ContainerStarted","Data":"717c1f5994100210430de032581077df2b6420a9d19d3b7234f39f48b477ee62"}
Nov 01 19:58:51.140911 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.140807    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qczmj" event={"ID":"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e","Type":"ContainerStarted","Data":"de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577"}
Nov 01 19:58:51.142053 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.142027    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jv26r" event={"ID":"baa37f05-fc26-4a6c-a194-e3b4a88a5941","Type":"ContainerStarted","Data":"02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2"}
Nov 01 19:58:51.812016 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.811930    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:58:51.812016 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.811981    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:58:51.816105 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.816080    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 19:58:51.817427 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.817405    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\""
Nov 01 19:58:51.817564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.817466    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 19:58:51.817564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.817554    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\""
Nov 01 19:58:51.817688 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:51.817627    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 19:58:52.040868 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:52.040833    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:52.137369 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:52.137266    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:52.149306 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:52.149228    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qczmj" event={"ID":"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e","Type":"ContainerStarted","Data":"987da532a8dbe845487606a4f6ce60d9f7a14e94dcab13b707c28823f77d94fb"}
Nov 01 19:58:53.154218 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.154174    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jv26r" event={"ID":"baa37f05-fc26-4a6c-a194-e3b4a88a5941","Type":"ContainerStarted","Data":"8fb2f3c33485a368f29bcc567bff09e52fa1868dc39e262d21fcf111f433a3d4"}
Nov 01 19:58:53.154763 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.154244    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jv26r" event={"ID":"baa37f05-fc26-4a6c-a194-e3b4a88a5941","Type":"ContainerStarted","Data":"d0ea4ba6018a0a13f20132bff97cb44890f7c83a5089717b8e3a49bdbe24863d"}
Nov 01 19:58:53.154763 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.154302    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jv26r"
Nov 01 19:58:53.172259 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.172203    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jv26r" podStartSLOduration=1.388323647 podStartE2EDuration="3.172187219s" totalImagesPullingTime="1.783863572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:53.171697617 +0000 UTC m=+50.910042664" watchObservedRunningTime="2026-11-01 19:58:53.172187219 +0000 UTC m=+50.910532263"
Nov 01 19:58:53.493589 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.493553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:53.560493 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.560457    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:53.746766 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.746670    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jv26r_baa37f05-fc26-4a6c-a194-e3b4a88a5941/dns/0.log"
Nov 01 19:58:53.946419 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:53.946391    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jv26r_baa37f05-fc26-4a6c-a194-e3b4a88a5941/kube-rbac-proxy/0.log"
Nov 01 19:58:54.146378 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:54.146297    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqvzs_8bf82705-43f4-4c30-a6f3-403bf0f0d899/dns-node-resolver/0.log"
Nov 01 19:58:54.157356 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:54.157329    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dr9z8" event={"ID":"34e81519-4820-4e98-976b-48933a0eb319","Type":"ContainerStarted","Data":"ab1301405ef408b057faa5d88bb18d9098963638e87e73cc9e534998a19f5cd7"}
Nov 01 19:58:54.159433 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:54.159406    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qczmj" event={"ID":"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e","Type":"ContainerStarted","Data":"37fce9d41fdb9f06b46e75e13dbb6e6659eec710e7208c8683fadd80776ad3ee"}
Nov 01 19:58:54.172510 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:54.172470    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dr9z8" podStartSLOduration=1.503781142 podStartE2EDuration="4.172458163s" totalImagesPullingTime="2.668677021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:54.171878573 +0000 UTC m=+51.910223624" watchObservedRunningTime="2026-11-01 19:58:54.172458163 +0000 UTC m=+51.910803214"
Nov 01 19:58:54.188268 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:54.188235    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qczmj" podStartSLOduration=1.836861211 podStartE2EDuration="4.188223487s" totalImagesPullingTime="2.351362276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:54.187924975 +0000 UTC m=+51.926270084" watchObservedRunningTime="2026-11-01 19:58:54.188223487 +0000 UTC m=+51.926568537"
Nov 01 19:58:54.468572 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:54.468532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:55.546279 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:55.546255    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skvhv_4ae3e006-540b-4bd3-b35e-7c3a9e1d130f/node-ca/0.log"
Nov 01 19:58:56.194391 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.194359    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.299996 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.299963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.349793 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.349765    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dr9z8_34e81519-4820-4e98-976b-48933a0eb319/serve-healthcheck-canary/0.log"
Nov 01 19:58:56.416905 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.416873    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"]
Nov 01 19:58:56.418923 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.418907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.422157 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.422129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.428440 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.428423    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 19:58:56.428680 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.428642    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 19:58:56.428680 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.428661    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g9z2t\""
Nov 01 19:58:56.428794 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.428704    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 19:58:56.428794 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.428717    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 19:58:56.428794 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.428766    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 19:58:56.431265 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.431246    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"]
Nov 01 19:58:56.434710 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.434683    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"]
Nov 01 19:58:56.436621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.436600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.450209 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.450148    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"]
Nov 01 19:58:56.455263 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.455241    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d5rd4"]
Nov 01 19:58:56.457288 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.457273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490279 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.490386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.490386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-sys\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78ph\" (UniqueName: \"kubernetes.io/projected/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-kube-api-access-k78ph\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.490521 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-tls\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490521 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-wtmp\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490521 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfh2\" (UniqueName: \"kubernetes.io/projected/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-kube-api-access-nbfh2\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490521 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.490635 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96e9e104-2216-4847-a03b-a6161ac495f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.490635 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-textfile\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490635 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.490772 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490772 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.490772 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.490909 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-root\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.490909 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7mj\" (UniqueName: \"kubernetes.io/projected/96e9e104-2216-4847-a03b-a6161ac495f2-kube-api-access-zm7mj\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.491011 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.491011 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-metrics-client-ca\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.491011 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.490982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.509007 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.508985    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 19:58:56.509226 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.509211    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 19:58:56.509441 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.509425    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 19:58:56.509492 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.509440    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4p76p\""
Nov 01 19:58:56.509492 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.509455    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 19:58:56.509603 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.509525    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-88k4l\""
Nov 01 19:58:56.527240 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.527212    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.527330 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.527258    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 19:58:56.527601 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.527584    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 19:58:56.591965 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.591935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-sys\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.591972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78ph\" (UniqueName: \"kubernetes.io/projected/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-kube-api-access-k78ph\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.591990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-tls\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-wtmp\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfh2\" (UniqueName: \"kubernetes.io/projected/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-kube-api-access-nbfh2\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-sys\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-wtmp\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96e9e104-2216-4847-a03b-a6161ac495f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-textfile\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592386 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-root\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7mj\" (UniqueName: \"kubernetes.io/projected/96e9e104-2216-4847-a03b-a6161ac495f2-kube-api-access-zm7mj\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-textfile\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96e9e104-2216-4847-a03b-a6161ac495f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-metrics-client-ca\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.592981 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.592936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.593456 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.593175    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.593517 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.593487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-root\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.593817 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.593790    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.594126 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.594015    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.594126 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.594115    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.594265 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.594206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-metrics-client-ca\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.595335 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.595298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-tls\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.595441 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.595422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.596113 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.596091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.596188 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.596145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.596372 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.596357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.596435 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.596421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.599309 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.599266    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfh2\" (UniqueName: \"kubernetes.io/projected/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-kube-api-access-nbfh2\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.600008 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.599986    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7mj\" (UniqueName: \"kubernetes.io/projected/96e9e104-2216-4847-a03b-a6161ac495f2-kube-api-access-zm7mj\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.600118 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.600103    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78ph\" (UniqueName: \"kubernetes.io/projected/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-kube-api-access-k78ph\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.631842 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.631820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.726828 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.726744    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.730683 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.730662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:58:56.748542 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.748515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:58:56.767811 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.767779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:58:56.827441 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.827414    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:56.927376 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:56.927342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:57.003285 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.003217    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"]
Nov 01 19:58:57.006012 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:57.005979    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aefdb60_6f0d_4d04_868c_3be0ef759d9d.slice/crio-069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e WatchSource:0}: Error finding container 069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e: Status 404 returned error can't find the container with id 069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e
Nov 01 19:58:57.027338 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.027316    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"]
Nov 01 19:58:57.040744 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:57.040718    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e9e104_2216_4847_a03b_a6161ac495f2.slice/crio-4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166 WatchSource:0}: Error finding container 4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166: Status 404 returned error can't find the container with id 4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166
Nov 01 19:58:57.169309 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.169276    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg" event={"ID":"96e9e104-2216-4847-a03b-a6161ac495f2","Type":"ContainerStarted","Data":"4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166"}
Nov 01 19:58:57.170474 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.170451    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5rd4" event={"ID":"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2","Type":"ContainerStarted","Data":"4dc3a6be000a2cdad9af106d4ecb8ad48f9079ae98e601c67d227f22ab315238"}
Nov 01 19:58:57.172280 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.172258    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf" event={"ID":"4aefdb60-6f0d-4d04-868c-3be0ef759d9d","Type":"ContainerStarted","Data":"b7e115e48b661a93bdf04c27fbd232c1eb46810a2d168c908a377b01cd40f5ff"}
Nov 01 19:58:57.172354 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.172308    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf" event={"ID":"4aefdb60-6f0d-4d04-868c-3be0ef759d9d","Type":"ContainerStarted","Data":"1db9c45bd87f732599e10b984326048349b9d9dfc30a9550638776a508c7cf21"}
Nov 01 19:58:57.172354 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.172339    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf" event={"ID":"4aefdb60-6f0d-4d04-868c-3be0ef759d9d","Type":"ContainerStarted","Data":"069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e"}
Nov 01 19:58:57.411481 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.411406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:57.485521 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.485492    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:58:57.487979 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.487952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.492355 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.492332    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 19:58:57.492482 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.492419    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 19:58:57.492482 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.492419    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 19:58:57.492482 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.492433    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 19:58:57.492482 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.492441    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 19:58:57.502078 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.502058    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 19:58:57.502078 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.502063    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 19:58:57.502198 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.502110    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 19:58:57.502198 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.502084    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 19:58:57.502198 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.502115    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-66gfv\""
Nov 01 19:58:57.517511 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.517485    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:58:57.519535 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.519494    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:57.602312 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602678 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602678 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602678 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602678 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602846 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602846 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602846 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.602846 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.603036 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.603036 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.603036 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.603036 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.602966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.610660 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.610623    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:57.703621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703621 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.703844 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.704144 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.704144 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.704144 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.704144 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.703975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.704566 ip-10-0-133-52 kubenswrapper[2576]: E1101 19:58:57.704544    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle podName:d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:58.204521188 +0000 UTC m=+55.942866415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208") : configmap references non-existent config key: ca-bundle.crt
Nov 01 19:58:57.705944 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.705911    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.706232 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.706207    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.707086 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.707052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.707917 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.707878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.708490 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.708445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.708761 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.708737    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.709218 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.709193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.710245 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.710217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.711543 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.711515    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:57.711972 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.711891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.715625 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.715599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.718103 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.718061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.726924 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.726903    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:57.811770 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:57.811737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:58.098860 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.098831    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:58:58.177916 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.177869    2576 generic.go:356] "Generic (PLEG): container finished" podID="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" containerID="e55272afe0e88c583d7e355d1bb71e3d0f3b8814aeb93787e1c22a15acc709f6" exitCode=0
Nov 01 19:58:58.178074 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.177966    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5rd4" event={"ID":"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2","Type":"ContainerDied","Data":"e55272afe0e88c583d7e355d1bb71e3d0f3b8814aeb93787e1c22a15acc709f6"}
Nov 01 19:58:58.209914 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.208744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:58.212242 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.211245    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:58.404938 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.404909    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:58:58.462569 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.462536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:58.548955 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.548832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:58.692475 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:58.692443    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:58:58.696641 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:58:58.696598    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43ad51e_23b4_4fb9_bb8b_b7e8cb7d5208.slice/crio-a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f WatchSource:0}: Error finding container a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f: Status 404 returned error can't find the container with id a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f
Nov 01 19:58:59.182902 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.182869    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f"}
Nov 01 19:58:59.184921 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.184890    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg" event={"ID":"96e9e104-2216-4847-a03b-a6161ac495f2","Type":"ContainerStarted","Data":"a05b06322884f38873783466354d63b9e21a3a38c313baf9a380ac74c52d089a"}
Nov 01 19:58:59.185034 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.184948    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg" event={"ID":"96e9e104-2216-4847-a03b-a6161ac495f2","Type":"ContainerStarted","Data":"786876cb3681774bd978570a2272a484ab2ad9f707ea86deb0a7bc538c2bb575"}
Nov 01 19:58:59.185034 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.184974    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg" event={"ID":"96e9e104-2216-4847-a03b-a6161ac495f2","Type":"ContainerStarted","Data":"8f66f00b7c7d1cae1f8a490c05b71cfedd3760fe04604cebcf443ae0361f6288"}
Nov 01 19:58:59.186914 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.186895    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5rd4" event={"ID":"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2","Type":"ContainerStarted","Data":"264e7ab7ec29ea1658f9c9cdaa6421dc35e943fce138f05c03af671490b7db71"}
Nov 01 19:58:59.187029 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.186929    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d5rd4" event={"ID":"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2","Type":"ContainerStarted","Data":"ae50d651554468bbcfed7a9ff81194daad430bf7c527257323dc477acf129118"}
Nov 01 19:58:59.188812 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.188792    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf" event={"ID":"4aefdb60-6f0d-4d04-868c-3be0ef759d9d","Type":"ContainerStarted","Data":"e1ba6545d1a2f9bcc3be95d4178de8fbd537024f8b22c748716849b8a738f84a"}
Nov 01 19:58:59.203637 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.203596    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg" podStartSLOduration=1.841810072 podStartE2EDuration="3.20358426s" totalImagesPullingTime="1.361774188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:59.202192108 +0000 UTC m=+56.940537159" watchObservedRunningTime="2026-11-01 19:58:59.20358426 +0000 UTC m=+56.941929310"
Nov 01 19:58:59.219178 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.219138    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf" podStartSLOduration=2.046084585 podStartE2EDuration="3.219127623s" totalImagesPullingTime="1.173043038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:59.218710949 +0000 UTC m=+56.957056000" watchObservedRunningTime="2026-11-01 19:58:59.219127623 +0000 UTC m=+56.957472673"
Nov 01 19:58:59.238084 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.238042    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d5rd4" podStartSLOduration=2.10532071 podStartE2EDuration="3.238031993s" totalImagesPullingTime="1.047263469s" totalInitContainerRuntime="85.447814ms" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:59.23740396 +0000 UTC m=+56.975749012" watchObservedRunningTime="2026-11-01 19:58:59.238031993 +0000 UTC m=+56.976377077"
Nov 01 19:58:59.370761 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.370727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:59.449691 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.449658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:58:59.592466 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:58:59.592428    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:00.193000 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.192923    2576 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c" exitCode=0
Nov 01 19:59:00.193372 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.193033    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"}
Nov 01 19:59:00.828698 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.828666    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b5895c7bb-p9lzh"]
Nov 01 19:59:00.830556 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.830531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.834009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.833991    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 19:59:00.834624 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.834607    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 19:59:00.841145 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.841123    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b5895c7bb-p9lzh"]
Nov 01 19:59:00.874862 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.874827    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7rc12t2atv6bv\""
Nov 01 19:59:00.874862 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.874837    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 19:59:00.875081 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.874896    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 19:59:00.875081 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.874904    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hv2vx\""
Nov 01 19:59:00.934511 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-client-certs\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.934694 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-metrics-server-audit-profiles\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.934694 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-client-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.934783 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48922500-8344-4233-92cf-23ad2ff2e844-audit-log\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.934783 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.934851 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjlt\" (UniqueName: \"kubernetes.io/projected/48922500-8344-4233-92cf-23ad2ff2e844-kube-api-access-gfjlt\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:00.934968 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:00.934946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-tls\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.035969 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.035923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036164 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjlt\" (UniqueName: \"kubernetes.io/projected/48922500-8344-4233-92cf-23ad2ff2e844-kube-api-access-gfjlt\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036229 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-tls\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036280 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-client-certs\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036280 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-metrics-server-audit-profiles\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036834 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-client-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036834 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48922500-8344-4233-92cf-23ad2ff2e844-audit-log\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036834 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48922500-8344-4233-92cf-23ad2ff2e844-audit-log\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.036834 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.036805    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.037575 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.037547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-metrics-server-audit-profiles\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.039301 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.039277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-client-certs\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.039405 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.039353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-tls\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.039764 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.039740    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-client-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.043702 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.043674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjlt\" (UniqueName: \"kubernetes.io/projected/48922500-8344-4233-92cf-23ad2ff2e844-kube-api-access-gfjlt\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.142244 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.142163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:01.223025 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.222993    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"]
Nov 01 19:59:01.225782 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.225759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.234376 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.234342    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"]
Nov 01 19:59:01.234814 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.234773    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 19:59:01.234925 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.234866    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p22kj\""
Nov 01 19:59:01.241824 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.241799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:01.333987 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.333945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:01.339061 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.339030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eea70939-6f2a-485c-8fc9-a335b62860be-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.339175 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.339076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69v98\" (UniqueName: \"kubernetes.io/projected/eea70939-6f2a-485c-8fc9-a335b62860be-kube-api-access-69v98\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.423891 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.423817    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b5895c7bb-p9lzh"]
Nov 01 19:59:01.426698 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:59:01.426665    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48922500_8344_4233_92cf_23ad2ff2e844.slice/crio-a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455 WatchSource:0}: Error finding container a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455: Status 404 returned error can't find the container with id a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455
Nov 01 19:59:01.439921 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.439899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eea70939-6f2a-485c-8fc9-a335b62860be-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.440009 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.439935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v98\" (UniqueName: \"kubernetes.io/projected/eea70939-6f2a-485c-8fc9-a335b62860be-kube-api-access-69v98\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.442443 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.442416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eea70939-6f2a-485c-8fc9-a335b62860be-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.447498 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.447473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v98\" (UniqueName: \"kubernetes.io/projected/eea70939-6f2a-485c-8fc9-a335b62860be-kube-api-access-69v98\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.546681 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.545538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:01.626749 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.626231    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"]
Nov 01 19:59:01.634133 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.631908    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.643882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.643679    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 19:59:01.643882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.643779    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 19:59:01.643882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.643842    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 19:59:01.644282 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.644105    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 19:59:01.644282 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.644160    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 19:59:01.644282 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.644230    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rw4rh\""
Nov 01 19:59:01.649583 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.649557    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 19:59:01.650949 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.650927    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"]
Nov 01 19:59:01.746884 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.746853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxcr\" (UniqueName: \"kubernetes.io/projected/37d28636-5a23-4fbd-88b3-6d2346c9ed50-kube-api-access-xfxcr\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747004 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.746906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747068 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.747002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-federate-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747068 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.747059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-serving-certs-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747167 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.747105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747167 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.747129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747167 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.747147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.747167 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.747164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-metrics-client-ca\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.836991 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.836964    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"]
Nov 01 19:59:01.840082 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:59:01.840052    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea70939_6f2a_485c_8fc9_a335b62860be.slice/crio-b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf WatchSource:0}: Error finding container b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf: Status 404 returned error can't find the container with id b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf
Nov 01 19:59:01.848191 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-serving-certs-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848283 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848283 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848283 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848283 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-metrics-client-ca\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848417 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxcr\" (UniqueName: \"kubernetes.io/projected/37d28636-5a23-4fbd-88b3-6d2346c9ed50-kube-api-access-xfxcr\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848417 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848479 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-federate-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.848987 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.848956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-serving-certs-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.849274 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.849250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-metrics-client-ca\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.849676 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.849632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.851185 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.851154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.851274 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.851235    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-federate-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.851904 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.851869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.851904 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.851888    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.858210 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.858187    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxcr\" (UniqueName: \"kubernetes.io/projected/37d28636-5a23-4fbd-88b3-6d2346c9ed50-kube-api-access-xfxcr\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:01.958020 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:01.957996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:02.208116 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.208081    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"}
Nov 01 19:59:02.208296 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.208158    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"}
Nov 01 19:59:02.208296 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.208196    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"}
Nov 01 19:59:02.208296 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.208230    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"}
Nov 01 19:59:02.208296 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.208264    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"}
Nov 01 19:59:02.209298 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.209271    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd" event={"ID":"eea70939-6f2a-485c-8fc9-a335b62860be","Type":"ContainerStarted","Data":"b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf"}
Nov 01 19:59:02.210743 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.210719    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh" event={"ID":"48922500-8344-4233-92cf-23ad2ff2e844","Type":"ContainerStarted","Data":"a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455"}
Nov 01 19:59:02.241368 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.241298    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"]
Nov 01 19:59:02.246804 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:59:02.246769    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d28636_5a23_4fbd_88b3_6d2346c9ed50.slice/crio-e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45 WatchSource:0}: Error finding container e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45: Status 404 returned error can't find the container with id e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45
Nov 01 19:59:02.251580 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.251493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:02.331136 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.331109    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:02.528182 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.527986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:02.673362 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.673327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:02.751706 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:02.751639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:03.161961 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:03.161877    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:03.217293 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:03.217248    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf" event={"ID":"37d28636-5a23-4fbd-88b3-6d2346c9ed50","Type":"ContainerStarted","Data":"e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45"}
Nov 01 19:59:03.229379 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:03.229341    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerStarted","Data":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"}
Nov 01 19:59:03.232913 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:03.232880    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh" event={"ID":"48922500-8344-4233-92cf-23ad2ff2e844","Type":"ContainerStarted","Data":"98e3ef451fd6545e9cda66966f286656d6bd9e55f1e16f653bc9825979172ac2"}
Nov 01 19:59:03.257270 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:03.256587    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.627386536 podStartE2EDuration="6.256569775s" totalImagesPullingTime="3.61655108s" totalInitContainerRuntime="12.632159ms" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:03.253369179 +0000 UTC m=+60.991714229" watchObservedRunningTime="2026-11-01 19:59:03.256569775 +0000 UTC m=+60.994914829"
Nov 01 19:59:03.272720 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:03.272667    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh" podStartSLOduration=1.674542267 podStartE2EDuration="3.272635059s" totalImagesPullingTime="1.598092792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:03.271892988 +0000 UTC m=+61.010238040" watchObservedRunningTime="2026-11-01 19:59:03.272635059 +0000 UTC m=+61.010980111"
Nov 01 19:59:04.238168 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.238141    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf" event={"ID":"37d28636-5a23-4fbd-88b3-6d2346c9ed50","Type":"ContainerStarted","Data":"288848cdf4d978ebf38f3eab26c0ebd7497acd913cd508400b534ee063d80c66"}
Nov 01 19:59:04.238308 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.238198    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf" event={"ID":"37d28636-5a23-4fbd-88b3-6d2346c9ed50","Type":"ContainerStarted","Data":"872bebf949009f077243478a0d9ef999a8f2fdb9bed0b3dfdb5387b70ae4261b"}
Nov 01 19:59:04.239509 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.239485    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd" event={"ID":"eea70939-6f2a-485c-8fc9-a335b62860be","Type":"ContainerStarted","Data":"311cb5a927abccd3945133c9c9e6951b02ae8be0707767c3216a7725a11d879d"}
Nov 01 19:59:04.240055 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.240034    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:04.244882 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.244854    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:04.254305 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.254268    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd" podStartSLOduration=1.530835843 podStartE2EDuration="3.254257184s" totalImagesPullingTime="1.723421341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:04.253436534 +0000 UTC m=+61.991781586" watchObservedRunningTime="2026-11-01 19:59:04.254257184 +0000 UTC m=+61.992602236"
Nov 01 19:59:04.600358 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.600323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:04.671045 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.670998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:04.908474 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.908397    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:04.982732 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:04.982700    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:05.244387 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:05.244352    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf" event={"ID":"37d28636-5a23-4fbd-88b3-6d2346c9ed50","Type":"ContainerStarted","Data":"a17cf5f3c6e1a5f2e49ffe6552ece448ccec63db1248fc14acd741d4bb4fb7ea"}
Nov 01 19:59:05.265964 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:05.265920    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf" podStartSLOduration=2.413660811 podStartE2EDuration="4.265906798s" totalImagesPullingTime="1.852245987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:05.263608563 +0000 UTC m=+63.001953614" watchObservedRunningTime="2026-11-01 19:59:05.265906798 +0000 UTC m=+63.004251848"
Nov 01 19:59:06.910775 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:06.910739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:07.014451 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:07.014416    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:08.510756 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.510719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:08.511119 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.510771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:08.513405 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.513386    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 19:59:08.513479 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.513465    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 19:59:08.523816 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.523785    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 19:59:08.523945 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.523828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:08.534137 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.534111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:08.611545 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.611510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:08.614391 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.614369    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 19:59:08.624564 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.624540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:08.629896 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.629878    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\""
Nov 01 19:59:08.635304 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.635284    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\""
Nov 01 19:59:08.638402 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.638379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:08.643578 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.643559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:08.825145 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.825078    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\""
Nov 01 19:59:08.833145 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.833121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:08.918838 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.918808    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmf5w"]
Nov 01 19:59:08.921760 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:08.921737    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wg7v7"]
Nov 01 19:59:08.922003 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:59:08.921977    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0c52c4_326c_4a61_ae5f_b589196d41e6.slice/crio-8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10 WatchSource:0}: Error finding container 8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10: Status 404 returned error can't find the container with id 8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10
Nov 01 19:59:08.926232 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:59:08.926212    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e7cc09_97e2_4003_8b99_ee00d3d8252a.slice/crio-f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b WatchSource:0}: Error finding container f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b: Status 404 returned error can't find the container with id f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b
Nov 01 19:59:09.102067 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:09.102013    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4jmtc"]
Nov 01 19:59:09.104710 ip-10-0-133-52 kubenswrapper[2576]: W1101 19:59:09.104679    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a98578_24d3_4c0b_ba53_d2285c9ed6bd.slice/crio-e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6 WatchSource:0}: Error finding container e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6: Status 404 returned error can't find the container with id e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6
Nov 01 19:59:09.263453 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:09.263411    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4jmtc" event={"ID":"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd","Type":"ContainerStarted","Data":"e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6"}
Nov 01 19:59:09.264747 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:09.264706    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wg7v7" event={"ID":"71e7cc09-97e2-4003-8b99-ee00d3d8252a","Type":"ContainerStarted","Data":"f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b"}
Nov 01 19:59:09.265883 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:09.265858    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmf5w" event={"ID":"8e0c52c4-326c-4a61-ae5f-b589196d41e6","Type":"ContainerStarted","Data":"8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10"}
Nov 01 19:59:09.509662 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:09.509620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:09.609488 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:09.609431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:11.280451 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.279577    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4jmtc" event={"ID":"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd","Type":"ContainerStarted","Data":"b90337683f385329d5717954d40dbf7162cddcdc0a5c941f2bee210ff06754e0"}
Nov 01 19:59:11.280451 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.279628    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4jmtc" event={"ID":"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd","Type":"ContainerStarted","Data":"1b15efd3da973c82c6b4c93f1ca3255bcb9925fd13fd207bbeaa3b21a7735a0e"}
Nov 01 19:59:11.302368 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.302108    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4jmtc" podStartSLOduration=67.148015194 podStartE2EDuration="1m8.302088899s" totalImagesPullingTime="1.154073705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:11.30079159 +0000 UTC m=+69.039136642" watchObservedRunningTime="2026-11-01 19:59:11.302088899 +0000 UTC m=+69.040433952"
Nov 01 19:59:11.330159 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.330118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:11.407863 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.407817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:11.775763 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.775724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:11.880368 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:11.880335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:12.004545 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:12.004507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:13.289827 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:13.289787    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wg7v7" event={"ID":"71e7cc09-97e2-4003-8b99-ee00d3d8252a","Type":"ContainerStarted","Data":"09544172617c3439a6c67850cc3c32a1b9c9ec5139872fe1f5964b10570db859"}
Nov 01 19:59:13.290239 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:13.289958    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:13.310292 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:13.310242    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wg7v7" podStartSLOduration=66.650270541 podStartE2EDuration="1m10.310225513s" totalImagesPullingTime="3.659954972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:13.310052256 +0000 UTC m=+71.048397308" watchObservedRunningTime="2026-11-01 19:59:13.310225513 +0000 UTC m=+71.048570564"
Nov 01 19:59:14.135588 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:14.135534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:14.235056 ip-10-0-133-52 kubenswrapper[2576]: I1101 19:59:14.235021    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:14.530396 ip-10-0-133-52 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 19:59:14.542078 ip-10-0-133-52 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 19:59:14.542247 ip-10-0-133-52 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 19:59:14.542298 ip-10-0-133-52 systemd[1]: kubelet.service: Consumed 4.294s CPU time.
Nov 01 19:59:14.553135 ip-10-0-133-52 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 19:59:14.615812 ip-10-0-133-52 kubenswrapper[6479]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:59:14.615812 ip-10-0-133-52 kubenswrapper[6479]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 19:59:14.615812 ip-10-0-133-52 kubenswrapper[6479]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:59:14.615812 ip-10-0-133-52 kubenswrapper[6479]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:59:14.617919 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617836    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.617919 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617919    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.617919 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617923    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617927    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617930    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617933    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617936    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617939    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617942    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617944    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617947    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617950    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617952    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617955    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617958    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617961    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617963    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617966    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617969    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617972    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617975    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617977    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.618020 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617986    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617989    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617992    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617994    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.617997    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618000    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618003    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618005    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618008    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618011    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618013    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618017    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618019    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618022    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618025    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618030    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618033    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618035    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618038    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618041    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.618609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618043    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618046    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618048    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618051    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618054    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618056    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618059    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618062    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618064    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618067    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618070    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618073    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618075    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618083    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618086    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618089    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618091    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618094    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618096    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.619411 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618100    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618102    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618105    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618108    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618111    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618114    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618116    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618120    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618127    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618130    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618135    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618138    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618141    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618144    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618146    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618149    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618152    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618155    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618157    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.619902 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618160    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618162    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618165    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618167    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618170    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618172    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618175    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618178    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618181    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618184    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618186    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618190    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618193    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618195    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618198    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618200    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618203    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618206    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618208    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.620365 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618211    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618213    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618219    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618222    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618224    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618227    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.618229    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620164    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620179    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620183    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620186    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620189    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620192    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620195    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620198    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620201    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620203    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620206    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620209    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620212    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.620888 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620214    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620217    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620220    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620222    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620225    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620228    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620231    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620234    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620237    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620239    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620242    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620245    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620247    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620250    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620252    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620255    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620258    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620260    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620263    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620272    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.621370 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620275    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620282    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620285    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620287    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620290    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620293    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620295    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620298    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620301    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620303    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620306    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620310    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620315    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620318    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620321    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620324    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620327    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620330    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620332    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.621882 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620335    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620338    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620340    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620343    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620345    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620348    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620350    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620353    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620356    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620359    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620362    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620365    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620368    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620371    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620373    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620376    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620379    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620382    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620384    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620387    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.622341 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620389    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620392    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620394    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620398    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620400    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620403    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620405    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620408    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620411    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620414    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620417    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620420    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620423    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620425    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620428    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620431    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620435    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620438    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620440    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.622829 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620443    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620446    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620449    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620452    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620454    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620457    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620460    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620462    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620465    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620468    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620470    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620473    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620475    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620478    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.620481    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621021    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621029    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621032    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621035    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621039    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.623287 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621042    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621045    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621047    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621050    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621054    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621056    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621060    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621062    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621064    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621067    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621069    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621072    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621075    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621077    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621080    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621082    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621085    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621087    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621090    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.623786 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621092    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621095    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621097    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621100    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621102    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621105    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621107    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621110    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621113    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621116    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621119    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621121    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621124    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621127    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621129    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621131    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621134    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621136    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621140    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.624235 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621143    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621145    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621148    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621150    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621153    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621156    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621158    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621161    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621163    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621166    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621168    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621171    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621174    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621176    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621179    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621181    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621184    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621186    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621189    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.624712 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621192    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621195    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621197    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621200    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621203    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621205    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621208    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621210    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621213    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621215    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621218    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621220    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621222    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621226    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621228    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621231    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621233    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621236    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621240    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621242    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.625164 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621245    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621247    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621250    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621252    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621255    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621257    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621259    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621262    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621265    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621267    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621271    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621275    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621278    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621285    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621288    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621291    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621294    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621297    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621299    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.625635 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621302    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621307    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621311    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621314    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.621317    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621418    6479 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621428    6479 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621439    6479 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621446    6479 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621451    6479 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621454    6479 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621461    6479 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621466    6479 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621469    6479 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621472    6479 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621476    6479 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621479    6479 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621483    6479 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621486    6479 flags.go:64] FLAG: --cgroup-root=""
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621489    6479 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621492    6479 flags.go:64] FLAG: --client-ca-file=""
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621494    6479 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621497    6479 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 19:59:14.626169 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621503    6479 flags.go:64] FLAG: --cluster-domain=""
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621506    6479 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621510    6479 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621516    6479 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621520    6479 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621524    6479 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621528    6479 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621531    6479 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621534    6479 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621537    6479 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621540    6479 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621543    6479 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621547    6479 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621550    6479 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621554    6479 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621557    6479 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621560    6479 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621564    6479 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621567    6479 flags.go:64] FLAG: --enable-server="true"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621570    6479 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621575    6479 flags.go:64] FLAG: --event-burst="100"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621578    6479 flags.go:64] FLAG: --event-qps="50"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621581    6479 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621584    6479 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 19:59:14.626731 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621587    6479 flags.go:64] FLAG: --eviction-hard=""
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621592    6479 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621595    6479 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621599    6479 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621602    6479 flags.go:64] FLAG: --eviction-soft=""
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621605    6479 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621608    6479 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621611    6479 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621615    6479 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621619    6479 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621622    6479 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621626    6479 flags.go:64] FLAG: --feature-gates=""
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621634    6479 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621638    6479 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621641    6479 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621665    6479 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621669    6479 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621672    6479 flags.go:64] FLAG: --help="false"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621675    6479 flags.go:64] FLAG: --hostname-override="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621679    6479 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621682    6479 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621685    6479 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621688    6479 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 19:59:14.627303 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621692    6479 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621694    6479 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621697    6479 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621701    6479 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621704    6479 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621707    6479 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621710    6479 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621713    6479 flags.go:64] FLAG: --kube-reserved=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621716    6479 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621719    6479 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621723    6479 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621725    6479 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621728    6479 flags.go:64] FLAG: --lock-file=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621731    6479 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621734    6479 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621737    6479 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621743    6479 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621748    6479 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621751    6479 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621754    6479 flags.go:64] FLAG: --logging-format="text"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621758    6479 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621761    6479 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621764    6479 flags.go:64] FLAG: --manifest-url=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621767    6479 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621772    6479 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 19:59:14.627886 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621775    6479 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621779    6479 flags.go:64] FLAG: --max-pods="110"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621782    6479 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621785    6479 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621788    6479 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621791    6479 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621794    6479 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621797    6479 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621800    6479 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621807    6479 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621810    6479 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621814    6479 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621817    6479 flags.go:64] FLAG: --pod-cidr=""
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621820    6479 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621824    6479 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621827    6479 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621829    6479 flags.go:64] FLAG: --port="10250"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621832    6479 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621835    6479 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a6b04bdb5645f0a4"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621839    6479 flags.go:64] FLAG: --qos-reserved=""
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621841    6479 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621844    6479 flags.go:64] FLAG: --register-node="true"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621847    6479 flags.go:64] FLAG: --register-with-taints=""
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621851    6479 flags.go:64] FLAG: --registry-burst="10"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621856    6479 flags.go:64] FLAG: --registry-qps="5"
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621859    6479 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 19:59:14.628487 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621861    6479 flags.go:64] FLAG: --reserved-memory=""
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621866    6479 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621870    6479 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621873    6479 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621876    6479 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621879    6479 flags.go:64] FLAG: --runonce="false"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621882    6479 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621885    6479 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621888    6479 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621891    6479 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621894    6479 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621897    6479 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621900    6479 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621904    6479 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621906    6479 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621909    6479 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621912    6479 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621915    6479 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621919    6479 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621922    6479 flags.go:64] FLAG: --system-cgroups=""
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621925    6479 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621930    6479 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621933    6479 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621936    6479 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621940    6479 flags.go:64] FLAG: --tls-min-version=""
Nov 01 19:59:14.629119 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621943    6479 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621946    6479 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621949    6479 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621952    6479 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621955    6479 flags.go:64] FLAG: --v="2"
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621961    6479 flags.go:64] FLAG: --version="false"
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621965    6479 flags.go:64] FLAG: --vmodule=""
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621969    6479 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.621973    6479 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622116    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622120    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622123    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622126    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622129    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622131    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622134    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622137    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622140    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622142    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622145    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622148    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622150    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.629719 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622153    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622156    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622158    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622161    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622165    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622168    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622170    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622173    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622177    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622181    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622185    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622188    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622191    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622194    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622196    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622200    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622203    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622205    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622210    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.630250 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622213    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622215    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622218    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622221    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622223    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622226    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622229    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622231    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622234    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622237    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622240    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622242    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622245    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622247    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622250    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622253    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622255    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622258    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622263    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.630822 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622266    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622269    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622272    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622275    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622278    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622280    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622283    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622286    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622288    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622293    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622295    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622298    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622302    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622304    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622307    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622310    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622312    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622315    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622317    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622320    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.631664 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622323    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622326    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622328    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622331    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622334    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622336    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622339    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622341    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622344    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622347    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622350    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622353    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622356    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622359    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622362    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622369    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622371    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622374    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622377    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.632470 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622379    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622382    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622386    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622389    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622391    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622395    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622398    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622401    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622403    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622406    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622409    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622411    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622414    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622417    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622420    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622423    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.633102 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.622451    6479 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622492    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622496    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622499    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622503    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622505    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622508    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622511    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622513    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622516    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622521    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622525    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622528    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622531    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622534    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622537    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622540    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622542    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622550    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.633518 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622552    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622555    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622557    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622561    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622564    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622567    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622570    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622573    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622576    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622578    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622581    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622584    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622586    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622589    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622592    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622594    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622597    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622599    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622602    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.634114 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622604    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622608    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622612    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622614    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622617    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622621    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622624    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622627    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622630    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622632    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622635    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622637    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622641    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622659    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622663    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622665    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622669    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622672    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622675    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.634609 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622677    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622680    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622684    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622686    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622689    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622692    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622694    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622697    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622700    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622702    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622705    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622707    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622710    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622713    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622716    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622718    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622721    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622723    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622726    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622729    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.635075 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622732    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622735    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622737    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622740    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622743    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622746    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622749    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622752    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622754    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622759    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622761    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622764    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622767    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622769    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622772    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622775    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622778    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622780    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622783    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622786    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.635631 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622789    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622791    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622794    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622797    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622799    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622802    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622805    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622808    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622810    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.622815    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.626467    6479 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.626481    6479 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.626498    6479 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.626503    6479 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626563    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626568    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626572    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626576    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626579    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626582    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.636217 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626585    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626587    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626590    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626593    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626595    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626598    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626601    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626604    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626607    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626609    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626612    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626615    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626619    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626623    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626626    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626629    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626631    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626634    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626637    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.636787 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626640    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626658    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626663    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626679    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626682    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626686    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626689    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626692    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626695    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626697    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626700    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626702    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626705    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626708    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626710    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626713    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626715    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626718    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626721    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626723    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.637242 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626726    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626728    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626731    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626734    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626737    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626739    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626742    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626745    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626749    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626752    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626754    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626757    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626759    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626762    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626765    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626768    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626770    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626773    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626776    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626779    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.637860 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626781    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626784    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626786    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626789    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626791    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626794    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626796    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626799    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626801    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626820    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626824    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626828    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626831    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626834    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626837    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626840    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626842    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626845    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626848    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626850    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.638331 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626853    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626856    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626858    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626861    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626863    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626866    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626868    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626871    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626874    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626876    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626880    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626884    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626887    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626890    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626893    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626895    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626898    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626900    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626903    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.638823 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626906    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.626908    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.626937    6479 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627071    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627076    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627080    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627083    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627086    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627089    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627092    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627095    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627098    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627101    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627103    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627106    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.639276 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627108    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627111    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627114    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627116    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627119    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627121    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627124    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627127    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627129    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627132    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627135    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627138    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627140    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627143    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627145    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627148    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627150    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627153    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627156    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627158    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.639658 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627160    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627163    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627166    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627168    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627171    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627173    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627176    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627178    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627181    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627183    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627186    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627188    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627191    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627193    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627196    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627198    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627201    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627203    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627206    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.640154 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627208    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627211    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627214    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627217    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627220    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627223    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627226    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627228    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627231    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627233    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627236    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627238    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627241    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627244    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627247    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627250    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627252    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627255    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627257    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627260    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.640632 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627262    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627265    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627268    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627270    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627273    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627277    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627280    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627283    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627286    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627289    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627292    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627295    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627299    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627302    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627305    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627309    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627311    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627315    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627318    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.641206 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627321    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627324    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627326    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627329    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627331    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627337    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627340    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627343    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627345    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627348    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627350    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627353    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627355    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627358    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627360    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627363    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.641794 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.627395    6479 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627428    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627432    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627435    6479 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627438    6479 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627441    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627443    6479 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627447    6479 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627450    6479 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627452    6479 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627455    6479 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627458    6479 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627460    6479 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627463    6479 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627466    6479 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627469    6479 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627472    6479 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627475    6479 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627477    6479 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627481    6479 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:59:14.642230 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627485    6479 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627489    6479 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627493    6479 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627495    6479 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627498    6479 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627502    6479 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627505    6479 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627508    6479 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627510    6479 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627513    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627516    6479 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627519    6479 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627521    6479 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627524    6479 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627526    6479 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627529    6479 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627532    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627535    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627538    6479 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:59:14.642737 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627541    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627543    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627546    6479 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627549    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627551    6479 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627554    6479 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627556    6479 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627559    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627561    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627564    6479 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627567    6479 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627570    6479 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627572    6479 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627575    6479 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627578    6479 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627581    6479 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627584    6479 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627587    6479 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627589    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:59:14.643194 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627592    6479 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627595    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627597    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627600    6479 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627602    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627605    6479 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627608    6479 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627610    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627613    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627615    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627620    6479 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627622    6479 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627625    6479 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627628    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627630    6479 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627633    6479 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627635    6479 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627638    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627640    6479 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627661    6479 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:59:14.643700 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627664    6479 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627667    6479 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627670    6479 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627673    6479 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627675    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627678    6479 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627680    6479 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627683    6479 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627687    6479 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627689    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627692    6479 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627695    6479 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627697    6479 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627700    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627702    6479 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627705    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627708    6479 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627710    6479 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627713    6479 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:59:14.644180 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627715    6479 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627718    6479 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627721    6479 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627725    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627727    6479 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627730    6479 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627734    6479 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627736    6479 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627739    6479 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:14.627741    6479 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.627794    6479 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.628734    6479 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.628785    6479 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.628961    6479 server.go:1014] "Starting client certificate rotation"
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.629100    6479 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.629149    6479 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:53:03 +0000 UTC" deadline="2028-06-18 17:37:41.488546728 +0000 UTC"
Nov 01 19:59:14.644629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.629177    6479 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14277h38m26.859371181s"
Nov 01 19:59:14.645044 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.629401    6479 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:59:14.645044 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.630437    6479 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:59:14.645044 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.633629    6479 log.go:25] "Validated CRI v1 runtime API"
Nov 01 19:59:14.645044 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.635534    6479 log.go:25] "Validated CRI v1 image API"
Nov 01 19:59:14.645044 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.636268    6479 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 19:59:14.645044 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.638395    6479 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8c64e93a-ec08-4559-8e06-95751b690061:/dev/nvme0n1p3 9baf065a-c77b-47e2-a340-35fd9130e602:/dev/nvme0n1p4]
Nov 01 19:59:14.645386 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.638411    6479 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dc3a6be000a2cdad9af106d4ecb8ad48f9079ae98e601c67d227f22ab315238/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dc3a6be000a2cdad9af106d4ecb8ad48f9079ae98e601c67d227f22ab315238/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6/userdata/shm major:0 minor:416 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b/userdata/shm major:0 minor:414 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07560b5d-9b60-4ba1-9811-c308393b9ada/volumes/kubernetes.io~projected/kube-api-access-mxg87:{mountpoint:/var/lib/kubelet/pods/07560b5d-9b60-4ba1-9811-c308393b9ada/volumes/kubernetes.io~projected/kube-api-access-mxg87 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a4efd09-675b-47ab-b4cc-c4da33b77d6a/volumes/kubernetes.io~projected/kube-api-access-6854b:{mountpoint:/var/lib/kubelet/pods/0a4efd09-675b-47ab-b4cc-c4da33b77d6a/volumes/kubernetes.io~projected/kube-api-access-6854b major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cf8f06e-9873-48b5-8f8a-29b93691922e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0cf8f06e-9873-48b5-8f8a-29b93691922e/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e81519-4820-4e98-976b-48933a0eb319/volumes/kubernetes.io~projected/kube-api-access-5m7v6:{mountpoint:/var/lib/kubelet/pods/34e81519-4820-4e98-976b-48933a0eb319/volumes/kubernetes.io~projected/kube-api-access-5m7v6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e81519-4820-4e98-976b-48933a0eb319/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/34e81519-4820-4e98-976b-48933a0eb319/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~projected/kube-api-access-xfxcr:{mountpoint:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~projected/kube-api-access-xfxcr major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~projected/kube-api-access-gfjlt:{mountpoint:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~projected/kube-api-access-gfjlt major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f/volumes/kubernetes.io~projected/kube-api-access-x94fk:{mountpoint:/var/lib/kubelet/pods/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f/volumes/kubernetes.io~projected/kube-api-access-x94fk major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~projected/kube-api-access-k78ph:{mountpoint:/var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~projected/kube-api-access-k78ph major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090/volumes/kubernetes.io~projected/kube-api-access-4nl2c:{mountpoint:/var/lib/kubelet/pods/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090/volumes/kubernetes.io~projected/kube-api-access-4nl2c major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71e7cc09-97e2-4003-8b99-ee00d3d8252a/volumes/kubernetes.io~projected/kube-api-access-2jnzh:{mountpoint:/var/lib/kubelet/pods/71e7cc09-97e2-4003-8b99-ee00d3d8252a/volumes/kubernetes.io~projected/kube-api-access-2jnzh major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bf82705-43f4-4c30-a6f3-403bf0f0d899/volumes/kubernetes.io~projected/kube-api-access-zp66t:{mountpoint:/var/lib/kubelet/pods/8bf82705-43f4-4c30-a6f3-403bf0f0d899/volumes/kubernetes.io~projected/kube-api-access-zp66t major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e0c52c4-326c-4a61-ae5f-b589196d41e6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8e0c52c4-326c-4a61-ae5f-b589196d41e6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/934b3987-3764-417a-8e86-353093bddc70/volumes/kubernetes.io~projected/kube-api-access-5qvcr:{mountpoint:/var/lib/kubelet/pods/934b3987-3764-417a-8e86-353093bddc70/volumes/kubernetes.io~projected/kube-api-access-5qvcr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~projected/kube-api-access-zm7mj:{mountpoint:/var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~projected/kube-api-access-zm7mj major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd/volumes/kubernetes.io~projected/kube-api-access-wsxgf:{mountpoint:/var/lib/kubelet/pods/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd/volumes/kubernetes.io~projected/kube-api-access-wsxgf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd/volumes/kubernetes.io~secret/metrics-certs major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volumes/kubernetes.io~projected/kube-api-access-6bdqf:{mountpoint:/var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volumes/kubernetes.io~projected/kube-api-access-6bdqf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baa37f05-fc26-4a6c-a194-e3b4a88a5941/volumes/kubernetes.io~projected/kube-api-access-m62v8:{mountpoint:/var/lib/kubelet/pods/baa37f05-fc26-4a6c-a194-e3b4a88a5941/volumes/kubernetes.io~projected/kube-api-access-m62v8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baa37f05-fc26-4a6c-a194-e3b4a88a5941/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/baa37f05-fc26-4a6c-a194-e3b4a88a5941/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/alertmanager/9 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~empty-dir/config-out major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~projected/kube-api-access-twgsz:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~projected/kube-api-access-twgsz major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~projected/tls-assets major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/config-volume major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/web-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/volumes/kubernetes.io~projected/kube-api-access-bp8dp:{mountpoint:/var/lib/kubelet/pods/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/volumes/kubernetes.io~projected/kube-api-access-bp8dp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~projected/kube-api-access-6prnc:{mountpoint:/var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~projected/kube-api-access-6prnc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eea70939-6f2a-485c-8fc9-a335b62860be/volumes/kubernetes.io~projected/kube-api-access-69v98:{mountpoint:/var/lib/kubelet/pods/eea70939-6f2a-485c-8fc9-a335b62860be/volumes/kubernetes.io~projected/kube-api-access-69v98 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eea70939-6f2a-485c-8fc9-a335b62860be/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/eea70939-6f2a-485c-8fc9-a335b62860be/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~projected/kube-api-access-nbfh2:{mountpoint:/var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~projected/kube-api-access-nbfh2 major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:251 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/eb91dce5e62f0af085aefd064a147ea70bc40143fa24c874d79392bc0a178f22/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2992fc2a8200dfc32898e344eb1c3807e693d1d349fcb0dbd8c47c41dc2dc4aa/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3d68390639af1e9f69c0868930cddd59cff83348ec2a910b54d60c0c1a0e3a65/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7cf4daa5e79d3e0837e2d41d597ece4343c81c0ee53b632c29a10d3474f1c9b6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d7680ab9fc6208a8c2cd5aacaeed98709c833ff28f8699e380aaafa1fb69866c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3f996e2548d194ad54d183d572fea2e4f4f038da536808d532fc03d43d5e28e3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6b79d8c631224c356f67dc67f56f6855be5fdfc8869072da211f0074fef7f15a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7a351f8eee3811cbcd0577f70dda7f6ebdf6e4e5bbc929d88782af1ad9d7577c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e7434b592f12afd8c6675395737d21729437915902011d015d0e298b9cd0c4a5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e14fe441f9f2fb47357ae3da483bf27403bf4170ba474f0bb61452b9b0f2ea24/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/f9b87d17d5d1c16b99283b0cf691a89fe9ff6a35b4683b34a85c4bdd1d039e04/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/0f7fd7a8e6711a04981d11343e0c273b581b72d7c4bf5e7081e66f0070b8fd29/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/99f04ef74b9fb9507087b55361e87e4115550435761d6dead2f8039b3f16ab49/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e2a86d703334da0b43c5c48df6baec8947cf654d969131bf1bdc0b8c85a9f8ce/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b64e96ae3c45d81e18037728150529caa20a094fe99ab41d00010f6108a4e54a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6db14992f3204ece1100fb1f0b6865e3810f974bfdf487710a6a65b6352098fc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8394df35bf451b989a487eb02e911dbfb8b1ad1ae6f2a5b6fa807c62ceaa7af5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f2cb7ec453f0076026597f43d05c3b306566eefc490ab8b3aa89492be697658c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d64e176340441df5a3f924c18e20066aa8730cb4c61061b08ee37f1e8c79b554/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e5654b48336f5ce35a1029d80b3288bb550a8df0687684d1b3cf01258e597ea3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8ab5def794f7587f7829e00dc18cd46f46ef2b5a0d7b2d3f4ff21115e8d8a605/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c6df6358c482fa6a22338d99ada46f97e0007994ff4fcdaed7e502b88150940e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8bd2eca239ac67d1c37b6c257a60e6c89ba4429898a2490ad5abf08c76cb8333/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/22f84b53c51b835dd56d416694ffea36deae8531ee1ae994ac8e7b6b4330b54a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9f10d1a8342608aa37ae91b380fbbc2a7e92c179b1a01e259ddcee13a3a8df1f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/71532145821efaa8489835cf775cfd141751c4f45962b5b5c1ab07b9bb8a54f8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/7a9b5d88abedba523b0f238850cc78162ec8eecffba879b7bfc4c7151cbe5f1b/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/c7d1ecf0148ebc64ab5dd9ef3502c25aa58138a420af2a8557b6a2106c5b0348/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/060f880045ab8de53aef7d3c53815305998d129846e4a5f7da90801fb3fa87af/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d977f4347c46ba4604121c1928ac66f0d24c586269bbd86f682f474392b5a7a2/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/59e389ff3cf6d94e63f52f1f4cbef4de057ab959fb11679286796984ee993259/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/e3dcc1df5567302e63991cabe52bc613434195b09a80758fedb26557dc40e79a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/c66b6dc65e8c9f92d54f2f691a97c8210cb5d6dc7180a6728f6bec1b1c5d51ec/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/6c4204462d0839e0ccb0865a2397fa0a8adcc63c1425240d747dc871bb412525/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/16771823d95355d3f5850696c07a94c0f296fe294f5771333d6388c3f6dfd25f/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/c68ecc5670417d579764f4ec747588ffd9f438f8b889a7d74e92746f37a70dae/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/0340e449305e06b4adebacc5ab645414598a36f42fbcd13830c534241991d85e/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/1b4fe21b2213ee5663c168d91974359af17db5837101a7e9f648c85223cf81f3/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/fedfd0d119ed76fabda32482fafdda16635a2bd64b6dc1ed456ac9ed15f29b03/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/a48ec15073e8094d745c15d4d326559a441e98785e0481c16178c57b2b7cb8f0/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/fcdc21739923d8ea3f14afe62f3902f7a733720e3b20063e601f0c35cc42d4c3/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a629a92e9ef90460d304d876f215d6ad3eefe4f1526e75a93f9ccaedc3db5566/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/c8c63a1c849365bd46bf189746b733ffc6ee6c38195d53f6a035b58295fc315e/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/4ee7aa3efd7b526092939c0b5954ef806900794398119677f1b28cec588d28fd/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/905197de6772af9bdb8cb4997ff0442d627ea515de2f80aad978e4b27fae670c/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/b1dd748807d6c5bf328cc8be2698814bf9c858c406f37e5b7b173c447efeade3/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/0bbbbe2e7eb52f794aabf29ac96a0db29ae3b30111d0c5f211bd2cedf535c81d/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/d809797a48d5b2a38ee140c04679ad6bd0cd6e60dff366a4e6ff57e7e33c1ca8/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/5671a6a86a5c9b7d83619670372a4dbe8bf53a512d506cc91ec427d2ecc2fa97/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/8d4811d6faba95a04d11b358e2a85ca8077342a8eeb3dfdd1c883fb4423e0011/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/0491a37361ee6a30b9e5b4357230ab7fbe03b2fc96925b7900a3538f68644421/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/6ed6f8da25b41d6c546e079e5b7ef55d073cd7347cdf9d4df7ac0920efc09371/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/ceeb8e58fc5e3deb62911b198e201249739131e7b111ecfdb2c8c86bf5823f6c/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/4a0240a8b10340725edd72c519678ae39be13429ed8579bf7bca4e24b2d8de12/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/6f6baa4e802fafd4e86245bb2e18623eb4b10c8f377257bc90eeee76539cdddb/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/9a86d4a654e64310da5b265a1cf90ad7577e59f34fab8f5b40f27dc7cabab415/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/1794dae54db13a6966040a84676967d21f00902ab01b9fe7564bc42db379f2f3/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/9458971ce58b2bc89c30f1857ee07df62e0941b43543fa8400d596ef8c3ca11c/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/e29f936624a638978fb7578a6170f3689494d363d658f658b0148414bcb66eff/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/8a630bfff25e79cfa8e34e1e2b7fc3c02e9478d8774f4b48aaaa863c0956b26d/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/d3ec46d715f0072eb19ca16f309414ee5d6ae4aebad874e65ab4810b91a48579/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/cc9e29295a352a96bbd2dba10373be17df389ce3b6cee5fe7b72a3f580b9f438/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/9d1ba59b2de86cc0b1ecb8c5cbda2b18a6d9045ea059b044aee91a5a0c79f2e4/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f18d33f81c3b5b6a4acc1ac2cc7fc71756c97baf73680b95b78f89da20f2e5d6/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7fde769fb6bea2a993dcf9ab65927d005f9e09cf0391a82237ae5d037514129a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/da0fb02840b03432b743e22b93fcd128c65c3b6bc98dc7bb67b541f532dfa389/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/196d9ed365d0367b0698714ca9b1b1f3adfb5a1876d9bccbb059b50e6a3c11ac/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7d1865f93b965eff6c9955b0cc4ee37cfb6dc2fbb32413f4211da3c81c91c6ac/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ac885cb77522262bdc2665e3a40815bce01b57148bb0ad7a14562a4d528ce5a4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ac376e041e5de8294a0c90d61d993b0c912861948154f54eb2cb4e7ad7191bbe/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8b8af0ed38f561625e30ddc3555e26db667cc6a90b2375f9f3067d5cb343bd0e/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6b5a160db3fe74953a2353aad7aa16aeedf18c4092963ffa0b83d4bee5875672/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fb07b2dd220c4ffba09df7f9cbe41df16fc54e3dcc318db6a227d18a10efbf76/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 19:59:14.646176 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.645787    6479 manager.go:217] Machine: {Timestamp:2026-11-01 19:59:14.645310212 +0000 UTC m=+0.060942675 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100250 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbdf649bb8347d93c5c2823ebb315 SystemUUID:ec2fbdf6-49bb-8347-d93c-5c2823ebb315 BootID:a54a3cd9-b05a-421b-971c-85683fa1a77f Filesystems:[{Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6125344658cb3fadf95b2a17da8d4a45ee6ca2a3d18be765e458a9cbf5d7bdf/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f/volumes/kubernetes.io~projected/kube-api-access-x94fk DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cf8f06e-9873-48b5-8f8a-29b93691922e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f4de34f70af88ec29ddf873c48814d078811b713fd178badf8f6ef6eea1a166/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baa37f05-fc26-4a6c-a194-e3b4a88a5941/volumes/kubernetes.io~projected/kube-api-access-m62v8 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~projected/kube-api-access-6prnc DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34e81519-4820-4e98-976b-48933a0eb319/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0fc198a182a022205d68ed6237b917c5fae903a5198f246a7f6a2e80790daf6/userdata/shm DeviceMajor:0 DeviceMinor:416 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~projected/kube-api-access-twgsz DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de99537601e923889554b16db6a49238bfe6c81ff53e29e40b2bcb406a816577/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a79c9c47275968d488c0823566f85eefc4b2fabbb96b66ff8fa934f7757b0455/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090/volumes/kubernetes.io~projected/kube-api-access-4nl2c DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98c4e7a3c6968baf1980ecdaa529033397a4d4bab056550321f5df985ffd9174/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02e1f3a2ca4c9319d1b0bf9b394a80986e1f8ab7881bb0788755b6c279fed6b2/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e913a002b30492c2a3f6978faaf52254e4367fe341d257580cf021623406cbc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/069468ede1737743fde11043cec95a7ecca016d1f711ee24669c7aa1a418505e/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dc3a6be000a2cdad9af106d4ecb8ad48f9079ae98e601c67d227f22ab315238/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eea70939-6f2a-485c-8fc9-a335b62860be/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/934b3987-3764-417a-8e86-353093bddc70/volumes/kubernetes.io~projected/kube-api-access-5qvcr DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71e7cc09-97e2-4003-8b99-ee00d3d8252a/volumes/kubernetes.io~projected/kube-api-access-2jnzh DeviceMajor:0 DeviceMinor:410 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1427c1f62e0a9f2037dac3e85c8017bd3169e6cff1d4e24b755a9615506c4e0/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60d2e67e20b49d157d2edfa5fc70225c9bb726304e10da9f8820f099b3fd14f4/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:406 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07560b5d-9b60-4ba1-9811-c308393b9ada/volumes/kubernetes.io~projected/kube-api-access-mxg87 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eea70939-6f2a-485c-8fc9-a335b62860be/volumes/kubernetes.io~projected/kube-api-access-69v98 DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd/volumes/kubernetes.io~projected/kube-api-access-wsxgf DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e0c52c4-326c-4a61-ae5f-b589196d41e6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:411 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f9004e4c82983953e8ead33d4badf4a9d216f89934a1b7224727a679c77ec10/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volumes/kubernetes.io~projected/kube-api-access-6bdqf DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b01e133a-c836-48dd-8c0b-aa317f79d00d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/volumes/kubernetes.io~projected/kube-api-access-bp8dp DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~projected/kube-api-access-zm7mj DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bf82705-43f4-4c30-a6f3-403bf0f0d899/volumes/kubernetes.io~projected/kube-api-access-zp66t DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a97b2a9772fd7b5bd69ef3f62b1d62cbc297493cf76e0585e2ad4a3098333d0/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96e9e104-2216-4847-a03b-a6161ac495f2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9c62288d186be698732fbd37bd0bf1b2bba8ffad9354afbf5a243484a59354b/userdata/shm DeviceMajor:0 DeviceMinor:414 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bcbb00ce97c8be652d30fd5ad3ad6eff0e8bd6cb0cdf1ffac7c780ee6252ca2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/baa37f05-fc26-4a6c-a194-e3b4a88a5941/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a4efd09-675b-47ab-b4cc-c4da33b77d6a/volumes/kubernetes.io~projected/kube-api-access-6854b DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~projected/kube-api-access-k78ph DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34e81519-4820-4e98-976b-48933a0eb319/volumes/kubernetes.io~projected/kube-api-access-5m7v6 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4aefdb60-6f0d-4d04-868c-3be0ef759d9d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37d28636-5a23-4fbd-88b3-6d2346c9ed50/volumes/kubernetes.io~projected/kube-api-access-xfxcr DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/989a62fb335d5e09c7d54cd0edf027572313f703e52645e6bb8d907a96722505/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e3e9cf3cd65c8f2fa60bb44162d22c3def8ce87051aa12a830741028c748902/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/volumes/kubernetes.io~projected/kube-api-access-nbfh2 DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~projected/kube-api-access-gfjlt DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2140d9ba2d2e7b1f93e27f4a7f7d145a62d27abfe9780f1bcaf6dd3d644ce974/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48922500-8344-4233-92cf-23ad2ff2e844/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e98431ea35da0a1b013b9ab88be1904677f914c5445a01d4ddbed7b471549b45/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02e1f3a2ca4c931 MacAddress:02:e9:30:00:61:eb Speed:10000 Mtu:8901} {Name:069468ede173774 MacAddress:62:a7:6a:71:eb:cb Speed:10000 Mtu:8901} {Name:4e3e9cf3cd65c8f MacAddress:92:9f:70:b5:3c:aa Speed:10000 Mtu:8901} {Name:4f4de34f70af88e MacAddress:ee:a2:26:0e:55:41 Speed:10000 Mtu:8901} {Name:8f9004e4c829839 MacAddress:3a:b6:21:4f:bb:44 Speed:10000 Mtu:8901} {Name:a79c9c47275968d MacAddress:62:56:e0:84:99:87 Speed:10000 Mtu:8901} {Name:a8fd0f58a3e9cd2 MacAddress:12:8a:74:42:83:c2 Speed:10000 Mtu:8901} {Name:b6125344658cb3f MacAddress:32:40:06:bd:50:4a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:ee:5d:43 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:61:91:74:5d:12 Speed:0 Mtu:8901} {Name:de99537601e9238 MacAddress:ae:8d:8b:cb:33:2c Speed:10000 Mtu:8901} {Name:e0fc198a182a022 MacAddress:6e:1b:b7:ad:8b:01 Speed:10000 Mtu:8901} {Name:e98431ea35da0a1 MacAddress:76:d6:ff:5a:15:be Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:ee:5d:43 Speed:0 Mtu:9001} {Name:f9c62288d186be6 MacAddress:ca:85:a9:a0:11:46 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4e:fe:ab:61:6e:ba Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:47:fb:ca:5a:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 19:59:14.646376 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646179    6479 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 19:59:14.646376 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646225    6479 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 19:59:14.646549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646520    6479 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 19:59:14.646714 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646552    6479 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-52.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 19:59:14.646756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646717    6479 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 19:59:14.646756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646726    6479 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 19:59:14.646756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646739    6479 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:59:14.646834 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646758    6479 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:59:14.646866 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.646851    6479 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 19:59:14.647024 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647015    6479 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 19:59:14.647134 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647125    6479 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 19:59:14.647172 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647165    6479 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 19:59:14.647200 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647181    6479 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 19:59:14.647200 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647195    6479 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 19:59:14.647253 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647203    6479 kubelet.go:405] "Adding apiserver pod source"
Nov 01 19:59:14.647253 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.647212    6479 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 19:59:14.648141 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.648121    6479 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 19:59:14.648141 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.648139    6479 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:59:14.650856 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.650523    6479 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 19:59:14.653621 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.653604    6479 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 19:59:14.653621 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.653620    6479 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:59:14.653862 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.653752    6479 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 19:59:14.654019 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654003    6479 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 19:59:14.654088 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654045    6479 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 19:59:14.654179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654167    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 19:59:14.654234 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654189    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 19:59:14.654234 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654199    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 19:59:14.654234 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654209    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 19:59:14.654234 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654218    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 19:59:14.654234 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654227    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654236    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654246    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654257    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654267    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654280    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654293    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654329    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 19:59:14.654468 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.654337    6479 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 19:59:14.657937 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.657921    6479 server.go:1263] "Started kubelet"
Nov 01 19:59:14.658048 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.657991    6479 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 19:59:14.660239 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.658033    6479 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 19:59:14.660239 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.658153    6479 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 19:59:14.660239 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.658579    6479 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 19:59:14.660239 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.659571    6479 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 19:59:14.658484 ip-10-0-133-52 systemd[1]: Started Kubernetes Kubelet.
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.663413    6479 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.663507    6479 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665288    6479 factory.go:55] Registering systemd factory
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665308    6479 factory.go:223] Registration of the systemd container factory successfully
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665398    6479 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665515    6479 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:53:05 +0000 UTC" deadline="2028-03-31 17:38:29.01165384 +0000 UTC"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665570    6479 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12381h39m14.34609093s"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665595    6479 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665758    6479 factory.go:153] Registering CRI-O factory
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665772    6479 factory.go:223] Registration of the crio container factory successfully
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665960    6479 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.665973    6479 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 19:59:14.666216 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.666146    6479 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 19:59:14.666898 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.666253    6479 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 19:59:14.666898 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.666282    6479 factory.go:103] Registering Raw factory
Nov 01 19:59:14.666898 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.666301    6479 manager.go:1199] Started watching for new ooms in manager
Nov 01 19:59:14.667969 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.667950    6479 manager.go:319] Starting recovery of all containers
Nov 01 19:59:14.670374 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.670348    6479 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:59:14.672450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672409    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07560b5d-9b60-4ba1-9811-c308393b9ada" volumeName="kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-cni-binary-copy" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672450    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-federate-client-tls" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672463    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/empty-dir/48922500-8344-4233-92cf-23ad2ff2e844-audit-log" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672475    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e0c52c4-326c-4a61-ae5f-b589196d41e6" volumeName="kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672483    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b01e133a-c836-48dd-8c0b-aa317f79d00d" volumeName="kubernetes.io/secret/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672492    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672499    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672507    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672520    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/projected/37d28636-5a23-4fbd-88b3-6d2346c9ed50-kube-api-access-xfxcr" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672530    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07560b5d-9b60-4ba1-9811-c308393b9ada" volumeName="kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-daemon-config" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672539    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4efd09-675b-47ab-b4cc-c4da33b77d6a" volumeName="kubernetes.io/configmap/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-iptables-alerter-script" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672548    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e81519-4820-4e98-976b-48933a0eb319" volumeName="kubernetes.io/projected/34e81519-4820-4e98-976b-48933a0eb319-kube-api-access-5m7v6" seLinuxMountContext=""
Nov 01 19:59:14.672549 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672555    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672566    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf82705-43f4-4c30-a6f3-403bf0f0d899" volumeName="kubernetes.io/projected/8bf82705-43f4-4c30-a6f3-403bf0f0d899-kube-api-access-zp66t" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672586    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e9e104-2216-4847-a03b-a6161ac495f2" volumeName="kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672599    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e9e104-2216-4847-a03b-a6161ac495f2" volumeName="kubernetes.io/projected/96e9e104-2216-4847-a03b-a6161ac495f2-kube-api-access-zm7mj" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672611    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b01e133a-c836-48dd-8c0b-aa317f79d00d" volumeName="kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-env-overrides" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672631    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e" volumeName="kubernetes.io/projected/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-api-access-bp8dp" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672658    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" volumeName="kubernetes.io/projected/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-kube-api-access-nbfh2" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672671    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baa37f05-fc26-4a6c-a194-e3b4a88a5941" volumeName="kubernetes.io/empty-dir/baa37f05-fc26-4a6c-a194-e3b4a88a5941-tmp-dir" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672684    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672696    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672707    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e81519-4820-4e98-976b-48933a0eb319" volumeName="kubernetes.io/secret/34e81519-4820-4e98-976b-48933a0eb319-cert" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672719    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baa37f05-fc26-4a6c-a194-e3b4a88a5941" volumeName="kubernetes.io/configmap/baa37f05-fc26-4a6c-a194-e3b4a88a5941-config-volume" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672737    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672753    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672764    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" volumeName="kubernetes.io/empty-dir/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-textfile" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672774    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" volumeName="kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672796    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672804    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cf8f06e-9873-48b5-8f8a-29b93691922e" volumeName="kubernetes.io/configmap/0cf8f06e-9873-48b5-8f8a-29b93691922e-konnectivity-ca" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672812    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672819    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-tls" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672828    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aefdb60-6f0d-4d04-868c-3be0ef759d9d" volumeName="kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672846    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed76efa0-e418-4a93-8b3f-5d95d7e5f33f" volumeName="kubernetes.io/projected/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-kube-api-access-6prnc" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672859    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-metrics-client-ca" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672872    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934b3987-3764-417a-8e86-353093bddc70" volumeName="kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672884    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e" volumeName="kubernetes.io/secret/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672892    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" volumeName="kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-metrics-client-ca" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672901    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07560b5d-9b60-4ba1-9811-c308393b9ada" volumeName="kubernetes.io/projected/07560b5d-9b60-4ba1-9811-c308393b9ada-kube-api-access-mxg87" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672913    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d28636-5a23-4fbd-88b3-6d2346c9ed50" volumeName="kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-client-tls" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672926    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672939    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672952    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baa37f05-fc26-4a6c-a194-e3b4a88a5941" volumeName="kubernetes.io/secret/baa37f05-fc26-4a6c-a194-e3b4a88a5941-metrics-tls" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672963    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65f03a9b-91cc-4cd0-b6a4-ad246a1ac090" volumeName="kubernetes.io/projected/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kube-api-access-4nl2c" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672976    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e9e104-2216-4847-a03b-a6161ac495f2" volumeName="kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-tls" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.672989    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b01e133a-c836-48dd-8c0b-aa317f79d00d" volumeName="kubernetes.io/projected/b01e133a-c836-48dd-8c0b-aa317f79d00d-kube-api-access-6bdqf" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673007    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed76efa0-e418-4a93-8b3f-5d95d7e5f33f" volumeName="kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-tmp" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673019    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4efd09-675b-47ab-b4cc-c4da33b77d6a" volumeName="kubernetes.io/projected/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-kube-api-access-6854b" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673030    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b01e133a-c836-48dd-8c0b-aa317f79d00d" volumeName="kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-config" seLinuxMountContext=""
Nov 01 19:59:14.673043 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673042    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baa37f05-fc26-4a6c-a194-e3b4a88a5941" volumeName="kubernetes.io/projected/baa37f05-fc26-4a6c-a194-e3b4a88a5941-kube-api-access-m62v8" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673070    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673082    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673092    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-client-ca-bundle" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673119    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71e7cc09-97e2-4003-8b99-ee00d3d8252a" volumeName="kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673133    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934b3987-3764-417a-8e86-353093bddc70" volumeName="kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673143    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934b3987-3764-417a-8e86-353093bddc70" volumeName="kubernetes.io/projected/934b3987-3764-417a-8e86-353093bddc70-kube-api-access-5qvcr" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673165    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e9e104-2216-4847-a03b-a6161ac495f2" volumeName="kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-metrics-client-ca" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673173    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673180    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cf8f06e-9873-48b5-8f8a-29b93691922e" volumeName="kubernetes.io/secret/0cf8f06e-9873-48b5-8f8a-29b93691922e-agent-certs" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673193    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673205    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ae3e006-540b-4bd3-b35e-7c3a9e1d130f" volumeName="kubernetes.io/projected/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-kube-api-access-x94fk" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673225    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e9e104-2216-4847-a03b-a6161ac495f2" volumeName="kubernetes.io/empty-dir/96e9e104-2216-4847-a03b-a6161ac495f2-volume-directive-shadow" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673246    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd" volumeName="kubernetes.io/projected/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-kube-api-access-wsxgf" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673263    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3a98578-24d3-4c0b-ba53-d2285c9ed6bd" volumeName="kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673279    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673297    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673324    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48922500-8344-4233-92cf-23ad2ff2e844" volumeName="kubernetes.io/projected/48922500-8344-4233-92cf-23ad2ff2e844-kube-api-access-gfjlt" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673337    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ae3e006-540b-4bd3-b35e-7c3a9e1d130f" volumeName="kubernetes.io/configmap/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-serviceca" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673358    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aefdb60-6f0d-4d04-868c-3be0ef759d9d" volumeName="kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673378    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf82705-43f4-4c30-a6f3-403bf0f0d899" volumeName="kubernetes.io/empty-dir/8bf82705-43f4-4c30-a6f3-403bf0f0d899-tmp-dir" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673394    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934b3987-3764-417a-8e86-353093bddc70" volumeName="kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-binary-copy" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673415    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673426    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e" volumeName="kubernetes.io/empty-dir/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-data-volume" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673438    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" volumeName="kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673457    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" volumeName="kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673468    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed76efa0-e418-4a93-8b3f-5d95d7e5f33f" volumeName="kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-tuned" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673482    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aefdb60-6f0d-4d04-868c-3be0ef759d9d" volumeName="kubernetes.io/configmap/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-metrics-client-ca" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673519    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e9e104-2216-4847-a03b-a6161ac495f2" volumeName="kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673532    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e" volumeName="kubernetes.io/configmap/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673546    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eea70939-6f2a-485c-8fc9-a335b62860be" volumeName="kubernetes.io/projected/eea70939-6f2a-485c-8fc9-a335b62860be-kube-api-access-69v98" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673565    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eea70939-6f2a-485c-8fc9-a335b62860be" volumeName="kubernetes.io/secret/eea70939-6f2a-485c-8fc9-a335b62860be-monitoring-plugin-cert" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673576    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" volumeName="kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-tls" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673586    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aefdb60-6f0d-4d04-868c-3be0ef759d9d" volumeName="kubernetes.io/projected/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-kube-api-access-k78ph" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673608    6479 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b01e133a-c836-48dd-8c0b-aa317f79d00d" volumeName="kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673628    6479 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 19:59:14.674201 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.673637    6479 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 19:59:14.676615 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.676585    6479 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 19:59:14.698191 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.698151    6479 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 19:59:14.699736 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.699536    6479 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 19:59:14.699736 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.699560    6479 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 19:59:14.699736 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.699582    6479 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 19:59:14.699736 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:14.699624    6479 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 19:59:14.702853 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.702830    6479 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:59:14.733076 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.733040    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c" exitCode=0
Nov 01 19:59:14.735324 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.735296    6479 generic.go:356] "Generic (PLEG): container finished" podID="f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2" containerID="e55272afe0e88c583d7e355d1bb71e3d0f3b8814aeb93787e1c22a15acc709f6" exitCode=0
Nov 01 19:59:14.743327 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.743298    6479 generic.go:356] "Generic (PLEG): container finished" podID="d5c237e4f4c3f7a42c5e7f09a4fd5922" containerID="fe87fe6355985d48991bc6fea94cf588e5e68f78edff9ef839fb64b22a4362ac" exitCode=0
Nov 01 19:59:14.757057 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.757029    6479 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="82ec516f9e6e7f864575305c4a6c3158d50e490f97e69ef66e79c6cbf284f894" exitCode=0
Nov 01 19:59:14.757057 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.757053    6479 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="42f9678475b0e5e5768ed24c4c961872b073968853ad0f06efc3ba24607025e1" exitCode=0
Nov 01 19:59:14.757057 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.757060    6479 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="0ebba3b1bfc2c0d6555f8560dcaaf7edafd49077cce69e640d260fccdad72116" exitCode=0
Nov 01 19:59:14.757247 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.757068    6479 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="4b6fd2cdbb5c58a9deae8eaf10f74eb08f89f326fcc179cce2d1c5bbfe6088ce" exitCode=0
Nov 01 19:59:14.757247 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.757074    6479 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="e955a95859bf09adb559ff3f53987938445a9e652abeb383129af1ce0f95c598" exitCode=0
Nov 01 19:59:14.757247 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.757079    6479 generic.go:356] "Generic (PLEG): container finished" podID="934b3987-3764-417a-8e86-353093bddc70" containerID="0e997eee7b852bddc0e6f001fd3cbe2f3bb8c190f1a59f78ab5c5f50c627fc40" exitCode=0
Nov 01 19:59:14.774283 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.774262    6479 manager.go:324] Recovery completed
Nov 01 19:59:14.798196 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798124    6479 cpu_manager.go:226] "Starting" policy="none"
Nov 01 19:59:14.798196 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798143    6479 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 19:59:14.798196 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798165    6479 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798304    6479 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798315    6479 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798332    6479 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798337    6479 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798343    6479 policy_none.go:50] "Start"
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798366    6479 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 19:59:14.798375 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798375    6479 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 19:59:14.798566 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798507    6479 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 19:59:14.798566 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798514    6479 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 19:59:14.798566 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.798519    6479 policy_none.go:44] "Start"
Nov 01 19:59:14.799757 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:14.799734    6479 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 19:59:14.803245 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.803212    6479 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 19:59:14.807087 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.807071    6479 manager.go:341] "Starting Device Plugin manager"
Nov 01 19:59:14.807185 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:14.807098    6479 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 19:59:14.807185 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.807108    6479 server.go:87] "Starting device plugin registration server"
Nov 01 19:59:14.808167 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808152    6479 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 19:59:14.808217 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808168    6479 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 19:59:14.808287 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808264    6479 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 19:59:14.808417 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808377    6479 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 19:59:14.808542 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808424    6479 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 19:59:14.808542 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808431    6479 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 19:59:14.808679 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.808507    6479 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T19:59:14.808413047Z","UUID":"802946de-4d4a-4875-bbbd-270d0a7310bd","Handler":null,"Name":"","Endpoint":""}
Nov 01 19:59:14.810095 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.810078    6479 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 19:59:14.810219 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.810205    6479 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 19:59:14.810320 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:14.810113    6479 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 19:59:14.909588 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.909539    6479 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:59:14.911747 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.911724    6479 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:59:14.911872 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.911761    6479 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:59:14.911872 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.911772    6479 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-52.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:59:14.911872 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.911813    6479 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:14.926579 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.926558    6479 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:14.926672 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:14.926619    6479 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.000752 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.000697    6479 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"]
Nov 01 19:59:15.001358 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001325    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"94566354f58b906840cf934237faff37","Type":"ContainerStarted","Data":"077f4e8b23014b2bf95d74c9ed6b77e6e27ecbb7f2baaa33037291a59c4b2db0"}
Nov 01 19:59:15.001403 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001365    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"94566354f58b906840cf934237faff37","Type":"ContainerStarted","Data":"f8851202bcf94ffa690491d6b561c2ded8087149f76734014e5c964c0b3d6efd"}
Nov 01 19:59:15.001438 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001419    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"d5c237e4f4c3f7a42c5e7f09a4fd5922","Type":"ContainerStarted","Data":"16237f23437c9ff3dfb89ef33698230823f6b7f4cea5f986d47299e2c7c4d18d"}
Nov 01 19:59:15.001467 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001437    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"d5c237e4f4c3f7a42c5e7f09a4fd5922","Type":"ContainerDied","Data":"fe87fe6355985d48991bc6fea94cf588e5e68f78edff9ef839fb64b22a4362ac"}
Nov 01 19:59:15.001467 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001456    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal" event={"ID":"d5c237e4f4c3f7a42c5e7f09a4fd5922","Type":"ContainerStarted","Data":"a0a059fcf117cac4cea68f32bb1da906cb46f83f41797b19cfc9c7fb87820141"}
Nov 01 19:59:15.001528 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001494    6479 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.001528 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.001437    6479 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.010839 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.010823    6479 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:59:15.010924 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:15.010869    6479 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.011389 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.011370    6479 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:59:15.011425 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:15.011408    6479 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.077835 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.077754    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94566354f58b906840cf934237faff37-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"94566354f58b906840cf934237faff37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.077835 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.077790    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.077835 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.077806    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.178838 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.178788    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94566354f58b906840cf934237faff37-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"94566354f58b906840cf934237faff37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.178838 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.178838    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.179028 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.178855    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.179028 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.178905    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.179028 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.178936    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c237e4f4c3f7a42c5e7f09a4fd5922-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"d5c237e4f4c3f7a42c5e7f09a4fd5922\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.179028 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.178915    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94566354f58b906840cf934237faff37-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" (UID: \"94566354f58b906840cf934237faff37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.647804 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.647771    6479 apiserver.go:52] "Watching apiserver"
Nov 01 19:59:15.656450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.656429    6479 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 19:59:15.657037 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.657016    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jtnpv","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-979sf","kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p","openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd","openshift-dns/dns-default-jv26r","openshift-dns/node-resolver-cqvzs","openshift-multus/multus-nd5rg","openshift-multus/network-metrics-daemon-4jmtc","openshift-ovn-kubernetes/ovnkube-node-rl4sb","openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg","openshift-cluster-node-tuning-operator/tuned-kqhqv","openshift-insights/insights-runtime-extractor-qczmj","openshift-monitoring/node-exporter-d5rd4","openshift-multus/multus-additional-cni-plugins-4ttgz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-wg7v7","openshift-network-operator/iptables-alerter-nl7sv","openshift-monitoring/metrics-server-b5895c7bb-p9lzh","openshift-image-registry/node-ca-skvhv","openshift-ingress-canary/ingress-canary-dr9z8","kube-system/global-pull-secret-syncer-cmf5w","openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"]
Nov 01 19:59:15.662464 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.662443    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.663134 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.663107    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.663348 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.663325    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 19:59:15.663427 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.663356    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.663498 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.663340    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.663609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.663591    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.664915 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.664894    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.665066 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665051    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.665232 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665211    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qrts\""
Nov 01 19:59:15.665445 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665422    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 19:59:15.665511 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665454    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.665511 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665053    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-j6fxj\""
Nov 01 19:59:15.665605 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665591    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbpdw\""
Nov 01 19:59:15.665720 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665703    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 19:59:15.665857 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665110    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.665923 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665714    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 19:59:15.666021 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666005    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 19:59:15.666079 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666019    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 19:59:15.666174 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666139    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.666312 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666295    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 19:59:15.666473 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666458    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 19:59:15.666535 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665051    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 19:59:15.666583 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666549    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.666629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666616    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgwx5\""
Nov 01 19:59:15.666706 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666611    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 19:59:15.666706 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666671    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8578\""
Nov 01 19:59:15.666818 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666474    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 19:59:15.666967 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.666950    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tdbxs\""
Nov 01 19:59:15.667021 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.667007    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\""
Nov 01 19:59:15.667100 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.665934    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6s44\""
Nov 01 19:59:15.667227 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.667089    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\""
Nov 01 19:59:15.667333 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.667299    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-24gff\""
Nov 01 19:59:15.667765 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.667747    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.670684 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.670663    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.670778 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.670767    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-754ws\""
Nov 01 19:59:15.670834 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.670778    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 19:59:15.670834 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.670829    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.671708 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.671688    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 19:59:15.671824 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.671767    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\""
Nov 01 19:59:15.671903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.671838    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgkqn\""
Nov 01 19:59:15.671969 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.671926    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 19:59:15.671969 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.671925    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 19:59:15.671969 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.671928    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 19:59:15.672156 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672016    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.672210 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672179    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 19:59:15.672273 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672211    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.672273 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672231    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9xfd7\""
Nov 01 19:59:15.672273 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672267    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.672459 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672362    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.672459 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672366    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 19:59:15.672459 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672419    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.672608 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672470    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.672608 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672593    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 19:59:15.672738 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.672709    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 19:59:15.673289 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673273    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 19:59:15.673613 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673594    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 19:59:15.673722 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673671    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 19:59:15.673722 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673717    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 19:59:15.673837 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673767    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 19:59:15.673837 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673774    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-88k4l\""
Nov 01 19:59:15.673966 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673869    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 19:59:15.673966 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.673891    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 19:59:15.674064 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674002    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 19:59:15.674064 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674000    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 19:59:15.674064 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674051    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9pms2\""
Nov 01 19:59:15.674064 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674051    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4p76p\""
Nov 01 19:59:15.674236 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674137    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 19:59:15.674668 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674633    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g9z2t\""
Nov 01 19:59:15.674862 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.674848    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 19:59:15.675071 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675053    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 19:59:15.675071 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675067    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 19:59:15.675202 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675057    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 19:59:15.675202 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675083    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 19:59:15.675202 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675103    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 19:59:15.675202 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675102    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 19:59:15.675202 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675162    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-66gfv\""
Nov 01 19:59:15.675202 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675162    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 19:59:15.675563 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675548    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 19:59:15.675638 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675563    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 19:59:15.675706 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675684    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 19:59:15.675706 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675693    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 19:59:15.675833 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675818    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hv2vx\""
Nov 01 19:59:15.675955 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.675943    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7rc12t2atv6bv\""
Nov 01 19:59:15.676351 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676337    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 19:59:15.676409 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676355    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 19:59:15.676409 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676394    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 19:59:15.676506 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676469    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 19:59:15.676719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676707    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p22kj\""
Nov 01 19:59:15.676719 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676717    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 19:59:15.676813 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676706    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rw4rh\""
Nov 01 19:59:15.676993 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.676959    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 19:59:15.677313 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.677301    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 19:59:15.677629 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.677613    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 19:59:15.678948 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.678907    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 19:59:15.681718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681699    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.681798 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681729    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-socket-dir-parent\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.681798 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681752    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:15.681798 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681779    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-node-log\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.681947 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681885    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.681947 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681924    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-registration-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.682050 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.681973    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94fk\" (UniqueName: \"kubernetes.io/projected/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-kube-api-access-x94fk\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.682050 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682009    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.682143 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682041    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvcr\" (UniqueName: \"kubernetes.io/projected/934b3987-3764-417a-8e86-353093bddc70-kube-api-access-5qvcr\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.682143 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682094    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-lib-modules\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.682233 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682147    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-metrics-certs\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:15.682233 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682157    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cf8f06e-9873-48b5-8f8a-29b93691922e-agent-certs\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:59:15.682233 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682215    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-dbus\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.682374 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682271    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.682374 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682293    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-netns\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.682374 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682335    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:15.682374 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682367    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-bin\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.682501 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682404    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cf8f06e-9873-48b5-8f8a-29b93691922e-agent-certs\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:59:15.682501 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682483    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nl2c\" (UniqueName: \"kubernetes.io/projected/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kube-api-access-4nl2c\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.682557 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682502    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-system-cni-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.682557 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682517    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.682557 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682532    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-multus\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.682557 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682550    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.682756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682566    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.682756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682674    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp66t\" (UniqueName: \"kubernetes.io/projected/8bf82705-43f4-4c30-a6f3-403bf0f0d899-kube-api-access-zp66t\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.682756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682704    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-kubelet-config\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.682756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682725    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-systemd-units\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.682756 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682746    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-slash\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682758    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682784    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdqf\" (UniqueName: \"kubernetes.io/projected/b01e133a-c836-48dd-8c0b-aa317f79d00d-kube-api-access-6bdqf\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682828    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-bin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682860    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-modprobe-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682893    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e0c52c4-326c-4a61-ae5f-b589196d41e6-original-pull-secret\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682899    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysconfig\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682940    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-var-lib-kubelet\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.682992 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.682974    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-conf-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683011    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-etc-kubernetes\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683033    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-kubelet\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683088    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-kubernetes\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683110    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683125    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-conf\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683155    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-system-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683172    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-k8s-cni-cncf-io\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683201    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-config\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683215    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-cnibin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.683264 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683245    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683323    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683368    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf82705-43f4-4c30-a6f3-403bf0f0d899-tmp-dir\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683412    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-config\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683509    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf82705-43f4-4c30-a6f3-403bf0f0d899-tmp-dir\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683560    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-socket-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683567    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/934b3987-3764-417a-8e86-353093bddc70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683605    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-cnibin\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683700    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-run\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683728    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-sys\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.683775 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683772    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxg87\" (UniqueName: \"kubernetes.io/projected/07560b5d-9b60-4ba1-9811-c308393b9ada-kube-api-access-mxg87\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683821    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-host-slash\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683875    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6854b\" (UniqueName: \"kubernetes.io/projected/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-kube-api-access-6854b\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.683927    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-device-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684034    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684063    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8bf82705-43f4-4c30-a6f3-403bf0f0d899-hosts-file\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684095    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-systemd\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684132    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-tmp\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684184    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-netd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684210    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-serviceca\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684227    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-tmp\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.684240 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684244    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-tuned\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684305    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prnc\" (UniqueName: \"kubernetes.io/projected/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-kube-api-access-6prnc\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684334    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-hostroot\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684374    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-tuned\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684398    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-var-lib-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684414    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684445    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-serviceca\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684454    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-cni-binary-copy\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684476    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-kubelet\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684502    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxgf\" (UniqueName: \"kubernetes.io/projected/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-kube-api-access-wsxgf\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684523    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-systemd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684539    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-log-socket\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.684609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684607    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-cni-binary-copy\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684617    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-env-overrides\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684685    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-sys-fs\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684717    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-host\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684735    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-netns\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684749    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-ovn\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684766    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-iptables-alerter-script\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684781    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684796    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovn-node-metrics-cert\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684801    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-env-overrides\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684832    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f06e-9873-48b5-8f8a-29b93691922e-konnectivity-ca\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684890    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-daemon-config\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684929    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-multus-certs\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684944    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-etc-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684973    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-script-lib\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.684988    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-host\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.685003    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cf8f06e-9873-48b5-8f8a-29b93691922e-konnectivity-ca\") pod \"konnectivity-agent-jtnpv\" (UID: \"0cf8f06e-9873-48b5-8f8a-29b93691922e\") " pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:59:15.685098 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.685035    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-iptables-alerter-script\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.685609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.685127    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovn-node-metrics-cert\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.685609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.685147    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-daemon-config\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.685609 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.685226    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b01e133a-c836-48dd-8c0b-aa317f79d00d-ovnkube-script-lib\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.688278 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.688256    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvcr\" (UniqueName: \"kubernetes.io/projected/934b3987-3764-417a-8e86-353093bddc70-kube-api-access-5qvcr\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.688349 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.688328    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnzh\" (UniqueName: \"kubernetes.io/projected/71e7cc09-97e2-4003-8b99-ee00d3d8252a-kube-api-access-2jnzh\") pod \"network-check-target-wg7v7\" (UID: \"71e7cc09-97e2-4003-8b99-ee00d3d8252a\") " pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:15.688349 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.688340    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94fk\" (UniqueName: \"kubernetes.io/projected/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-kube-api-access-x94fk\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.688434 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.688358    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nl2c\" (UniqueName: \"kubernetes.io/projected/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kube-api-access-4nl2c\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.688848 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.688829    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp66t\" (UniqueName: \"kubernetes.io/projected/8bf82705-43f4-4c30-a6f3-403bf0f0d899-kube-api-access-zp66t\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.688919 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.688873    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdqf\" (UniqueName: \"kubernetes.io/projected/b01e133a-c836-48dd-8c0b-aa317f79d00d-kube-api-access-6bdqf\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.689049 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.689032    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6854b\" (UniqueName: \"kubernetes.io/projected/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-kube-api-access-6854b\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.689185 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.689168    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prnc\" (UniqueName: \"kubernetes.io/projected/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-kube-api-access-6prnc\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.689422 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.689406    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxg87\" (UniqueName: \"kubernetes.io/projected/07560b5d-9b60-4ba1-9811-c308393b9ada-kube-api-access-mxg87\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.692246 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.692230    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxgf\" (UniqueName: \"kubernetes.io/projected/a3a98578-24d3-4c0b-ba53-d2285c9ed6bd-kube-api-access-wsxgf\") pod \"network-metrics-daemon-4jmtc\" (UID: \"a3a98578-24d3-4c0b-ba53-d2285c9ed6bd\") " pod="openshift-multus/network-metrics-daemon-4jmtc"
Nov 01 19:59:15.762944 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.762914    6479 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.763113 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.762998    6479 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.768406 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.768385    6479 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 19:59:15.774896 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.774880    6479 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:59:15.774992 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:15.774924    6479 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.778053 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.778039    6479 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:59:15.778094 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:15.778073    6479 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.us-east-2.compute.internal"
Nov 01 19:59:15.786073 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786053    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-multus\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.786179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786080    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.786179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786099    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.786179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786116    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78ph\" (UniqueName: \"kubernetes.io/projected/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-kube-api-access-k78ph\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.786179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786142    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48922500-8344-4233-92cf-23ad2ff2e844-audit-log\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.786179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786165    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-slash\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.786179 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786170    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786188    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786175    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-multus\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786229    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786369    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-slash\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786372    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48922500-8344-4233-92cf-23ad2ff2e844-audit-log\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786423    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.786450 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786430    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786485    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786460    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-client-certs\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786530    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v98\" (UniqueName: \"kubernetes.io/projected/eea70939-6f2a-485c-8fc9-a335b62860be-kube-api-access-69v98\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786557    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-cnibin\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786586    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786656    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-cnibin\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.786718 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786690    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786732    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786737    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-client-certs\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786764    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-conf\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786794    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786823    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-cnibin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786852    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786853    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786891    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786912    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-cnibin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786917    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786927    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-conf\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786951    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-socket-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786981    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.786991    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787008    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-run\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787033    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-host-slash\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787050    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.787047 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787059    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-device-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787053    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-socket-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787087    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8bf82705-43f4-4c30-a6f3-403bf0f0d899-hosts-file\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787093    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-run\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787116    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787145    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-data-volume\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787133    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787167    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8bf82705-43f4-4c30-a6f3-403bf0f0d899-hosts-file\") pod \"node-resolver-cqvzs\" (UID: \"8bf82705-43f4-4c30-a6f3-403bf0f0d899\") " pod="openshift-dns/node-resolver-cqvzs"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787152    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-device-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787118    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4efd09-675b-47ab-b4cc-c4da33b77d6a-host-slash\") pod \"iptables-alerter-nl7sv\" (UID: \"0a4efd09-675b-47ab-b4cc-c4da33b77d6a\") " pod="openshift-network-operator/iptables-alerter-nl7sv"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787173    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-root\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787190    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787249    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-data-volume\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787284    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-systemd\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787310    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787318    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-metrics-client-ca\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787325    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-systemd\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787373    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787374    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787421    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m7v6\" (UniqueName: \"kubernetes.io/projected/34e81519-4820-4e98-976b-48933a0eb319-kube-api-access-5m7v6\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787449    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787478    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787484    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-metrics-client-ca\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787511    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/baa37f05-fc26-4a6c-a194-e3b4a88a5941-config-volume\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787534    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787546    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787549    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787553    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-kubelet\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787451    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787592    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-kubelet\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787782    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/baa37f05-fc26-4a6c-a194-e3b4a88a5941-config-volume\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787797    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787848    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787834    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96e9e104-2216-4847-a03b-a6161ac495f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787898    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-host\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787927    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96e9e104-2216-4847-a03b-a6161ac495f2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787929    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-netns\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787970    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-ovn\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787972    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-netns\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.787971 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.787993    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788022    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788026    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-host\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788050    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-multus-certs\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788050    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788074    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-etc-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788083    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-ovn\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788104    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-host\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788139    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3e006-540b-4bd3-b35e-7c3a9e1d130f-host\") pod \"node-ca-skvhv\" (UID: \"4ae3e006-540b-4bd3-b35e-7c3a9e1d130f\") " pod="openshift-image-registry/node-ca-skvhv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788143    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-serving-certs-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788193    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-registration-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788203    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788224    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-etc-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788224    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7mj\" (UniqueName: \"kubernetes.io/projected/96e9e104-2216-4847-a03b-a6161ac495f2-kube-api-access-zm7mj\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788259    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-registration-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788276    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788330    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788195    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-multus-certs\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788353    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788260    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-client-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788385    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-crio-socket\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788402    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-dbus\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788415    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-netns\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788434    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-serving-certs-ca-bundle\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788471    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-system-cni-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788499    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-client-ca-bundle\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788506    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-dbus\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788514    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-netns\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788502    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788547    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788563    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjlt\" (UniqueName: \"kubernetes.io/projected/48922500-8344-4233-92cf-23ad2ff2e844-kube-api-access-gfjlt\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788578    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-metrics-client-ca\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788598    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-kubelet-config\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788525    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-system-cni-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788674    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-systemd-units\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788704    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788710    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-systemd-units\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788674    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e0c52c4-326c-4a61-ae5f-b589196d41e6-kubelet-config\") pod \"global-pull-secret-syncer-cmf5w\" (UID: \"8e0c52c4-326c-4a61-ae5f-b589196d41e6\") " pod="kube-system/global-pull-secret-syncer-cmf5w"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788740    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788746    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788787    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-sys\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788792    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37d28636-5a23-4fbd-88b3-6d2346c9ed50-metrics-client-ca\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788815    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-bin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788853    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-modprobe-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788878    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysconfig\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788885    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-var-lib-cni-bin\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788906    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788901    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-var-lib-kubelet\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788940    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-var-lib-kubelet\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788951    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-conf-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788960    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-modprobe-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788969    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-etc-kubernetes\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788928    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788950    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysconfig\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789003    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-conf-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.788851    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789042    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-kubelet\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789045    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-etc-kubernetes\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789064    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e81519-4820-4e98-976b-48933a0eb319-cert\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789075    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-kubelet\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789101    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-kubernetes\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789149    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789165    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-kubernetes\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789240    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-system-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789256    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-k8s-cni-cncf-io\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789274    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34e81519-4820-4e98-976b-48933a0eb319-cert\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789279    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa37f05-fc26-4a6c-a194-e3b4a88a5941-tmp-dir\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789309    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-host-run-k8s-cni-cncf-io\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789309    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-tls\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789338    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789345    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-system-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789353    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789394    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-etc-sysctl-d\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789417    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baa37f05-fc26-4a6c-a194-e3b4a88a5941-tmp-dir\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789434    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.789428 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789461    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789484    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-federate-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789509    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-sys\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789538    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789567    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789574    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789591    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfh2\" (UniqueName: \"kubernetes.io/projected/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-kube-api-access-nbfh2\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789618    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-metrics-server-audit-profiles\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789623    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-sys\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789658    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-netd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789685    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-textfile\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789712    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-hostroot\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789741    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/934b3987-3764-417a-8e86-353093bddc70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ttgz\" (UID: \"934b3987-3764-417a-8e86-353093bddc70\") " pod="openshift-multus/multus-additional-cni-plugins-4ttgz"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789728    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/37d28636-5a23-4fbd-88b3-6d2346c9ed50-federate-client-tls\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789575    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48922500-8344-4233-92cf-23ad2ff2e844-secret-metrics-server-tls\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789688    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789509    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789763    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-hostroot\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789820    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789813    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-netd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789837    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-textfile\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789857    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-var-lib-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789883    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789908    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789916    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-var-lib-openvswitch\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789948    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.789982    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-systemd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790020    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-log-socket\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790034    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-run-systemd\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790045    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-sys-fs\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790051    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48922500-8344-4233-92cf-23ad2ff2e844-metrics-server-audit-profiles\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790087    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-log-socket\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790087    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790090    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-sys-fs\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790544    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxcr\" (UniqueName: \"kubernetes.io/projected/37d28636-5a23-4fbd-88b3-6d2346c9ed50-kube-api-access-xfxcr\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790583    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8dp\" (UniqueName: \"kubernetes.io/projected/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-api-access-bp8dp\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.790618    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62v8\" (UniqueName: \"kubernetes.io/projected/baa37f05-fc26-4a6c-a194-e3b4a88a5941-kube-api-access-m62v8\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.791160    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.791249    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-wtmp\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.791287    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-socket-dir-parent\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.791357    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-socket-dir-parent\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.791888    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792006    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-node-log\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792042    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792073    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/baa37f05-fc26-4a6c-a194-e3b4a88a5941-metrics-tls\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792102    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-lib-modules\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792140    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-tls\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792181    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eea70939-6f2a-485c-8fc9-a335b62860be-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792211    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792239    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-bin\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792280    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792601    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96e9e104-2216-4847-a03b-a6161ac495f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792677    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-node-log\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792735    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65f03a9b-91cc-4cd0-b6a4-ad246a1ac090-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvs9p\" (UID: \"65f03a9b-91cc-4cd0-b6a4-ad246a1ac090\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvs9p"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.792996    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/baa37f05-fc26-4a6c-a194-e3b4a88a5941-metrics-tls\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.793136    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed76efa0-e418-4a93-8b3f-5d95d7e5f33f-lib-modules\") pod \"tuned-kqhqv\" (UID: \"ed76efa0-e418-4a93-8b3f-5d95d7e5f33f\") " pod="openshift-cluster-node-tuning-operator/tuned-kqhqv"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.793452    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-tls\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.793743    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eea70939-6f2a-485c-8fc9-a335b62860be-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.793833    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07560b5d-9b60-4ba1-9811-c308393b9ada-multus-cni-dir\") pod \"multus-nd5rg\" (UID: \"07560b5d-9b60-4ba1-9811-c308393b9ada\") " pod="openshift-multus/multus-nd5rg"
Nov 01 19:59:15.793913 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.793882    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01e133a-c836-48dd-8c0b-aa317f79d00d-host-cni-bin\") pod \"ovnkube-node-rl4sb\" (UID: \"b01e133a-c836-48dd-8c0b-aa317f79d00d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:15.798263 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.798242    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjlt\" (UniqueName: \"kubernetes.io/projected/48922500-8344-4233-92cf-23ad2ff2e844-kube-api-access-gfjlt\") pod \"metrics-server-b5895c7bb-p9lzh\" (UID: \"48922500-8344-4233-92cf-23ad2ff2e844\") " pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:15.798539 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.798518    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m7v6\" (UniqueName: \"kubernetes.io/projected/34e81519-4820-4e98-976b-48933a0eb319-kube-api-access-5m7v6\") pod \"ingress-canary-dr9z8\" (UID: \"34e81519-4820-4e98-976b-48933a0eb319\") " pod="openshift-ingress-canary/ingress-canary-dr9z8"
Nov 01 19:59:15.798573 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.798520    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78ph\" (UniqueName: \"kubernetes.io/projected/4aefdb60-6f0d-4d04-868c-3be0ef759d9d-kube-api-access-k78ph\") pod \"openshift-state-metrics-7bd7489957-979sf\" (UID: \"4aefdb60-6f0d-4d04-868c-3be0ef759d9d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-979sf"
Nov 01 19:59:15.798573 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.798524    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7mj\" (UniqueName: \"kubernetes.io/projected/96e9e104-2216-4847-a03b-a6161ac495f2-kube-api-access-zm7mj\") pod \"kube-state-metrics-5dc8d7d89b-pc6wg\" (UID: \"96e9e104-2216-4847-a03b-a6161ac495f2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pc6wg"
Nov 01 19:59:15.799553 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.799534    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v98\" (UniqueName: \"kubernetes.io/projected/eea70939-6f2a-485c-8fc9-a335b62860be-kube-api-access-69v98\") pod \"monitoring-plugin-77f4f76594-f4xxd\" (UID: \"eea70939-6f2a-485c-8fc9-a335b62860be\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:15.801406 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.801376    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8dp\" (UniqueName: \"kubernetes.io/projected/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-kube-api-access-bp8dp\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.802567 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.802549    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62v8\" (UniqueName: \"kubernetes.io/projected/baa37f05-fc26-4a6c-a194-e3b4a88a5941-kube-api-access-m62v8\") pod \"dns-default-jv26r\" (UID: \"baa37f05-fc26-4a6c-a194-e3b4a88a5941\") " pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:15.802706 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.802629    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfh2\" (UniqueName: \"kubernetes.io/projected/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-kube-api-access-nbfh2\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.802849 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.802832    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") pod \"alertmanager-main-0\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:15.803152 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.803128    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxcr\" (UniqueName: \"kubernetes.io/projected/37d28636-5a23-4fbd-88b3-6d2346c9ed50-kube-api-access-xfxcr\") pod \"telemeter-client-b8fbbf98d-ddfvf\" (UID: \"37d28636-5a23-4fbd-88b3-6d2346c9ed50\") " pod="openshift-monitoring/telemeter-client-b8fbbf98d-ddfvf"
Nov 01 19:59:15.893078 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893041    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-wtmp\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893089    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-root\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893116    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-crio-socket\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893134    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-sys\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893163    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-root\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893189    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-sys\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893172    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2-node-exporter-wtmp\") pod \"node-exporter-d5rd4\" (UID: \"f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2\") " pod="openshift-monitoring/node-exporter-d5rd4"
Nov 01 19:59:15.893250 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:15.893176    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e-crio-socket\") pod \"insights-runtime-extractor-qczmj\" (UID: \"ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e\") " pod="openshift-insights/insights-runtime-extractor-qczmj"
Nov 01 19:59:17.436313 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:17.436282    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:17.636839 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:17.636806    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jv26r"
Nov 01 19:59:19.768192 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:19.768155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:19.875448 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:19.875410    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:19.975786 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:19.975753    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:20.734692 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:20.734663    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:20.990224 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:20.990120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:21.132031 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:21.131996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:21.956193 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:21.956162    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:21.956433 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:21.956407    6479 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:59:23.029341 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:23.029308    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f4xxd"
Nov 01 19:59:23.263350 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:23.263309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:23.331605 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:23.331518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:26.369271 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:26.369175    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b5895c7bb-p9lzh"
Nov 01 19:59:26.770128 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:26.770066    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:26.868612 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:26.868576    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:26.966891 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:26.966856    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:29.168105 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:29.168068    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 19:59:31.140736 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.140697    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:59:31.141373 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.141328    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-metric" containerID="cri-o://002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33" gracePeriod=120
Nov 01 19:59:31.141373 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.141338    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="config-reloader" containerID="cri-o://bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac" gracePeriod=120
Nov 01 19:59:31.141567 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.141365    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-web" containerID="cri-o://8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473" gracePeriod=120
Nov 01 19:59:31.141567 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.141381    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy" containerID="cri-o://cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c" gracePeriod=120
Nov 01 19:59:31.141567 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.141401    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="alertmanager" containerID="cri-o://d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a" gracePeriod=120
Nov 01 19:59:31.141567 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.141401    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="prom-label-proxy" containerID="cri-o://aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58" gracePeriod=120
Nov 01 19:59:31.382419 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.382395    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:31.516517 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516484    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516517 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516521    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516774 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516554    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516774 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516593    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516774 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516633    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516774 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516686    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516774 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516732    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.516774 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516767    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.517075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516805    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.517075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516862    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.517075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516895    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.517075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516925    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.517075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516949    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:31.517075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.516971    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") pod \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\" (UID: \"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208\") "
Nov 01 19:59:31.517371 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.517183    6479 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.517857 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.517826    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:31.518176 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.518149    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:31.520283 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.520252    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.520709 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.520629    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "kube-api-access-twgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:31.520929 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.520895    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.521006 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.520901    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:31.521071 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.521005    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:31.521071 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.521058    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.521205 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.521182    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.521301 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.521277    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.523961 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.523822    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.530057 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.530035    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config" pod "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" (UID: "d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:31.618440 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618411    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-metrics-client-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618440 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618438    6479 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618450    6479 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-main-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618460    6479 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-cluster-tls-config\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618470    6479 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618478    6479 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-config-out\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618487    6479 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618495    6479 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-alertmanager-main-db\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618503    6479 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-tls-assets\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618532    6479 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-web-config\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618544    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgsz\" (UniqueName: \"kubernetes.io/projected/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-kube-api-access-twgsz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.618585 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.618552    6479 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:31.821001 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.820918    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58" exitCode=0
Nov 01 19:59:31.821001 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.820950    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33" exitCode=0
Nov 01 19:59:31.821001 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.820959    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c" exitCode=0
Nov 01 19:59:31.821001 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.820967    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473" exitCode=0
Nov 01 19:59:31.821001 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.820975    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac" exitCode=0
Nov 01 19:59:31.821001 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.820983    6479 generic.go:356] "Generic (PLEG): container finished" podID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a" exitCode=0
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821003    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"}
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821031    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821080    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"}
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821108    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"}
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821143    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"}
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821152    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821169    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"}
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821193    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"}
Nov 01 19:59:31.821260 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.821222    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208","Type":"ContainerDied","Data":"a8fd0f58a3e9cd2eba20d6090a465e464a186425fedf63c2f6857d0db215939f"}
Nov 01 19:59:31.828852 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.828835    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.835543 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.835522    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.841877 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.841859    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.844345 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.844323    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:59:31.848764 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.848750    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.852742 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.852717    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:59:31.856686 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.856665    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.863498 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.863482    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.870274 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.870245    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.870546 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.870514    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.870637 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.870554    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"} err="failed to get container status \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist"
Nov 01 19:59:31.870637 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.870595    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.870862 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.870846    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.870906 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.870868    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"} err="failed to get container status \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist"
Nov 01 19:59:31.870906 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.870882    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.871150 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.871127    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.871196 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871157    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"} err="failed to get container status \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist"
Nov 01 19:59:31.871196 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871174    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.871411 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.871389    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.871462 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871422    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"} err="failed to get container status \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist"
Nov 01 19:59:31.871462 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871436    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.871714 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.871696    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.871766 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871719    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"} err="failed to get container status \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist"
Nov 01 19:59:31.871766 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871733    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.871967 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.871947    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.872003 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871970    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"} err="failed to get container status \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist"
Nov 01 19:59:31.872003 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.871983    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.872198 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:31.872184    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.872236 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872202    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"} err="failed to get container status \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist"
Nov 01 19:59:31.872236 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872216    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.872415 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872400    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"} err="failed to get container status \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist"
Nov 01 19:59:31.872458 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872415    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.872631 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872613    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"} err="failed to get container status \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist"
Nov 01 19:59:31.872631 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872631    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.872865 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872849    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"} err="failed to get container status \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist"
Nov 01 19:59:31.872922 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.872865    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.873069 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873053    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"} err="failed to get container status \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist"
Nov 01 19:59:31.873167 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873068    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.873292 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873276    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"} err="failed to get container status \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist"
Nov 01 19:59:31.873333 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873291    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.873494 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873478    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"} err="failed to get container status \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist"
Nov 01 19:59:31.873494 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873493    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.873738 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873719    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"} err="failed to get container status \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist"
Nov 01 19:59:31.873795 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873740    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.873966 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873950    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"} err="failed to get container status \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist"
Nov 01 19:59:31.874014 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.873965    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.874174 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874153    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"} err="failed to get container status \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist"
Nov 01 19:59:31.874223 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874175    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.874411 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874389    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"} err="failed to get container status \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist"
Nov 01 19:59:31.874451 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874411    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.874613 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874589    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"} err="failed to get container status \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist"
Nov 01 19:59:31.874675 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874614    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.874862 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874841    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"} err="failed to get container status \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist"
Nov 01 19:59:31.874902 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.874862    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.875054 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875040    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"} err="failed to get container status \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist"
Nov 01 19:59:31.875101 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875053    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.875243 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875226    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"} err="failed to get container status \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist"
Nov 01 19:59:31.875286 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875244    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.875447 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875425    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"} err="failed to get container status \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist"
Nov 01 19:59:31.875499 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875449    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.875903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875873    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"} err="failed to get container status \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist"
Nov 01 19:59:31.876059 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.875904    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.876289    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"} err="failed to get container status \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.876314    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.876567    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"} err="failed to get container status \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.876597    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.876844    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"} err="failed to get container status \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.876864    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877085    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"} err="failed to get container status \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877111    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877348    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"} err="failed to get container status \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877367    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877604    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"} err="failed to get container status \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877620    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877860    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"} err="failed to get container status \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.877875    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878081    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"} err="failed to get container status \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878096    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878285    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"} err="failed to get container status \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878303    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878517    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"} err="failed to get container status \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878535    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878735    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"} err="failed to get container status \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.878751    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879047    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"} err="failed to get container status \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879064    6479 scope.go:122] "RemoveContainer" containerID="aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879253    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58"} err="failed to get container status \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": rpc error: code = NotFound desc = could not find container \"aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58\": container with ID starting with aa005b49995e6108e8091526eb5d728dd560090bcad8a0c255524086b3bc5e58 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879267    6479 scope.go:122] "RemoveContainer" containerID="002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879464    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33"} err="failed to get container status \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": rpc error: code = NotFound desc = could not find container \"002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33\": container with ID starting with 002cc7b1b31212bfdf475395373de8d46d0b5e0274be63ddcd73ffe16ce50e33 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879479    6479 scope.go:122] "RemoveContainer" containerID="cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879684    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c"} err="failed to get container status \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": rpc error: code = NotFound desc = could not find container \"cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c\": container with ID starting with cbd98e9500fba87d682d012e512693811bf52f4f656abc6d9cba4de27334238c not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879702    6479 scope.go:122] "RemoveContainer" containerID="8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879946    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473"} err="failed to get container status \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": rpc error: code = NotFound desc = could not find container \"8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473\": container with ID starting with 8fc8321418b326c6879d048b7a429754314fabd1029a894df3d43f589fa44473 not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.879966    6479 scope.go:122] "RemoveContainer" containerID="bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.880197    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac"} err="failed to get container status \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": rpc error: code = NotFound desc = could not find container \"bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac\": container with ID starting with bab2d4d109ef4cbe0b3cdec9e28fad54bb580650e4791ced4f553a2a62ace3ac not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.880218    6479 scope.go:122] "RemoveContainer" containerID="d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.880412    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a"} err="failed to get container status \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": rpc error: code = NotFound desc = could not find container \"d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a\": container with ID starting with d6f28017e2736c69c957d49f48d52c2e824c323c5bb3adda1dccc2057eb1db7a not found: ID does not exist"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.880427    6479 scope.go:122] "RemoveContainer" containerID="e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"
Nov 01 19:59:31.880953 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.880595    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c"} err="failed to get container status \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": rpc error: code = NotFound desc = could not find container \"e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c\": container with ID starting with e1bf7c38fab297be29e03edcc76f16561b41e1d35a522c07e0c9c2747e21ca2c not found: ID does not exist"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881565    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881828    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="prom-label-proxy"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881839    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="prom-label-proxy"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881851    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-metric"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881856    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-metric"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881873    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="config-reloader"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881878    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="config-reloader"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881884    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881894    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881900    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="alertmanager"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881904    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="alertmanager"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881912    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-web"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881919    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-web"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881933    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="init-config-reloader"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881937    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="init-config-reloader"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881991    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.881999    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="alertmanager"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.882004    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-web"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.882010    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="init-config-reloader"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.882015    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="prom-label-proxy"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.882020    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="kube-rbac-proxy-metric"
Nov 01 19:59:31.882603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.882028    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" containerName="config-reloader"
Nov 01 19:59:31.888590 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.888568    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:31.897916 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.897893    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 19:59:31.898014 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.897928    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 19:59:31.898014 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.897948    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 19:59:31.898014 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.897989    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 19:59:31.898132 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.898047    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 19:59:31.898132 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.898052    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 19:59:31.898132 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.898110    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 19:59:31.898277 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.898167    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-66gfv\""
Nov 01 19:59:31.898277 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.898172    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 19:59:31.898389 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.898351    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 19:59:31.899492 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:31.899469    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:59:32.022066 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022028    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aba37058-37a4-48fe-bfc2-12cdca9b241a-config-out\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022066 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022069    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022292 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022099    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022292 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022219    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022292 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022252    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba37058-37a4-48fe-bfc2-12cdca9b241a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022292 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022279    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aba37058-37a4-48fe-bfc2-12cdca9b241a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022493 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022392    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022493 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022432    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aba37058-37a4-48fe-bfc2-12cdca9b241a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022493 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022479    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aba37058-37a4-48fe-bfc2-12cdca9b241a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022559    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-config-volume\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022603 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022588    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-web-config\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022707 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022619    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.022707 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.022642    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br68z\" (UniqueName: \"kubernetes.io/projected/aba37058-37a4-48fe-bfc2-12cdca9b241a-kube-api-access-br68z\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123299 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123214    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aba37058-37a4-48fe-bfc2-12cdca9b241a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123299 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123255    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123299 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123277    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aba37058-37a4-48fe-bfc2-12cdca9b241a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123584 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123396    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aba37058-37a4-48fe-bfc2-12cdca9b241a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123584 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123451    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-config-volume\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123584 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123476    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-web-config\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123584 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123522    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123584 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123547    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br68z\" (UniqueName: \"kubernetes.io/projected/aba37058-37a4-48fe-bfc2-12cdca9b241a-kube-api-access-br68z\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123584 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123583    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aba37058-37a4-48fe-bfc2-12cdca9b241a-config-out\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123608    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123640    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123703    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123732    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba37058-37a4-48fe-bfc2-12cdca9b241a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.123903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123816    6479 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 19:59:32.123903 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.123895    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aba37058-37a4-48fe-bfc2-12cdca9b241a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.124231 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.124203    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aba37058-37a4-48fe-bfc2-12cdca9b241a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.125238 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.125208    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba37058-37a4-48fe-bfc2-12cdca9b241a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.126546 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.126514    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aba37058-37a4-48fe-bfc2-12cdca9b241a-config-out\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.126857 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.126830    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.126951 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.126864    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.126951 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.126837    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.126951 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.126941    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.127075 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.127000    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.127120 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.127106    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aba37058-37a4-48fe-bfc2-12cdca9b241a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.127771 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.127755    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-web-config\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.128251 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.128229    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aba37058-37a4-48fe-bfc2-12cdca9b241a-config-volume\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.131566 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.131547    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br68z\" (UniqueName: \"kubernetes.io/projected/aba37058-37a4-48fe-bfc2-12cdca9b241a-kube-api-access-br68z\") pod \"alertmanager-main-0\" (UID: \"aba37058-37a4-48fe-bfc2-12cdca9b241a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.207354 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.207306    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 19:59:32.487168 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.487136    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 19:59:32.490773 ip-10-0-133-52 kubenswrapper[6479]: W1101 19:59:32.490731    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba37058_37a4_48fe_bfc2_12cdca9b241a.slice/crio-a0beaa8ddaf602dd460677e3023c57523fbd6f8ee2e81bdf4fa0db884c90f413 WatchSource:0}: Error finding container a0beaa8ddaf602dd460677e3023c57523fbd6f8ee2e81bdf4fa0db884c90f413: Status 404 returned error can't find the container with id a0beaa8ddaf602dd460677e3023c57523fbd6f8ee2e81bdf4fa0db884c90f413
Nov 01 19:59:32.704117 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.704074    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208" path="/var/lib/kubelet/pods/d43ad51e-23b4-4fb9-bb8b-b7e8cb7d5208/volumes"
Nov 01 19:59:32.826568 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.826483    6479 generic.go:356] "Generic (PLEG): container finished" podID="aba37058-37a4-48fe-bfc2-12cdca9b241a" containerID="2e2ac5b66bf63b1780bb1cf1946c7c1cfecef99bb368622fe8d46a62ecf25d49" exitCode=0
Nov 01 19:59:32.826771 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.826568    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerDied","Data":"2e2ac5b66bf63b1780bb1cf1946c7c1cfecef99bb368622fe8d46a62ecf25d49"}
Nov 01 19:59:32.826771 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:32.826667    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"a0beaa8ddaf602dd460677e3023c57523fbd6f8ee2e81bdf4fa0db884c90f413"}
Nov 01 19:59:33.295352 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.295318    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:33.374720 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.374637    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:33.668919 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.668836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:33.745213 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.745179    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:33.834960 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.834921    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"faac3cb355d038823b322abc17317f71d7c75c1d476e57bcf37d4d557748fa71"}
Nov 01 19:59:33.835147 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.835026    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"0b2d1c0f0084f73a14e3bd0d221823ff73287926e6fb0b9eca59a6a228e37163"}
Nov 01 19:59:33.835147 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.835066    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"e99705f6a2d7b4cc528ab1cca8247e82e5117f25161ac38583a23115cb4bfd8e"}
Nov 01 19:59:33.835147 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.835130    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"a2bc7a75f2014011db0b0955438b90b32ff16e1225418fa458cb9ca45b1a49a7"}
Nov 01 19:59:33.835309 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.835166    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"c4be912688ad1e9746d0a9b4cb569e9f214705e4e9b5c947bd0334da89831304"}
Nov 01 19:59:33.835309 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.835206    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aba37058-37a4-48fe-bfc2-12cdca9b241a","Type":"ContainerStarted","Data":"5cb01188ec9f28eb8b0f472d1cba714b181755097da0a111dcfef5606288d9be"}
Nov 01 19:59:33.861750 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:33.861672    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.849655887 podStartE2EDuration="2.861660557s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.00467ms" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:33.860113767 +0000 UTC m=+19.275746242" watchObservedRunningTime="2026-11-01 19:59:33.861660557 +0000 UTC m=+19.277293021"
Nov 01 19:59:38.777074 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:38.777040    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:38.847161 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:38.847131    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:41.656587 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:41.656552    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rl4sb"
Nov 01 19:59:41.776620 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:41.776574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:41.877180 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:41.877147    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:41.978351 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:41.978320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:42.209626 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:42.209579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:42.277120 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:42.277059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 19:59:44.550045 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:44.550018    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jtnpv"
Nov 01 19:59:47.439086 ip-10-0-133-52 kubenswrapper[6479]: I1101 19:59:47.439055    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wg7v7"
Nov 01 19:59:59.168713 ip-10-0-133-52 kubenswrapper[6479]: E1101 19:59:59.168675    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:00:21.034381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:00:21.034349    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:00:29.170108 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:00:29.168252    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:00:31.312634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:00:31.312601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:00:49.663459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:00:49.663424    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:00:52.765008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:00:52.764977    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:00:52.834797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:00:52.834761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:00:59.168918 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:00:59.168835    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:01:00.775669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:00.775615    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:00.891801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:00.891769    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:01.018985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.018952    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:01.713102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.713070    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"]
Nov 01 20:01:01.716370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.716351    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.721545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.721514    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-55bvj\""
Nov 01 20:01:01.721672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.721558    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 20:01:01.721672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.721600    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 20:01:01.726800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.726776    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"]
Nov 01 20:01:01.803592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.803553    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.804001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.803607    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.804001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.803740    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjfh\" (UniqueName: \"kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.904817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.904776    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjfh\" (UniqueName: \"kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.905004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.904849    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.905004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.904877    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.905264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.905249    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.905301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.905251    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:01.912853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:01.912824    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjfh\" (UniqueName: \"kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:02.033089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.033001    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:02.070487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.070451    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"]
Nov 01 20:01:02.073952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.073927    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.082248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.081205    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 20:01:02.082248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.081244    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 20:01:02.082248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.081317    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 20:01:02.084803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.084722    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:02.094264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.094191    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"]
Nov 01 20:01:02.099727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.099431    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 20:01:02.099727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.099586    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbwtw\""
Nov 01 20:01:02.099727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.099667    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 20:01:02.099727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.099731    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 20:01:02.107177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107152    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.107177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107181    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm2h\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.107389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107217    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.107389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107314    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.107389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107332    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.107389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107353    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.107389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.107371    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.177799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.177768    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:02.208007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.207974    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208013    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208045    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208068    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208113    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208131    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm2h\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208156    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.208800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.208774    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.210551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.210524    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.210667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.210569    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.210915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.210897    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.211004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.210981    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.215640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.215619    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.216476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.216454    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm2h\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h\") pod \"istiod-openshift-gateway-7876bfcb74-hgh74\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.303881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.303794    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"]
Nov 01 20:01:02.307198 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:01:02.307165    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c131fc_30cf_4216_844f_18c779b82926.slice/crio-96dd2bfaf87ae95e44bd8b5564b77780451c6797ccd63bed87096e58a56a41b4 WatchSource:0}: Error finding container 96dd2bfaf87ae95e44bd8b5564b77780451c6797ccd63bed87096e58a56a41b4: Status 404 returned error can't find the container with id 96dd2bfaf87ae95e44bd8b5564b77780451c6797ccd63bed87096e58a56a41b4
Nov 01 20:01:02.308995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.308980    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:01:02.389174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.389142    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:02.662243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:02.662173    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"]
Nov 01 20:01:02.665869 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:01:02.665837    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72595b16_b70f_447e_bba2_1043475ffd59.slice/crio-e05b38edccadf79078fc1083927257a18f0e80ff1441ac04eabeff24f97e923e WatchSource:0}: Error finding container e05b38edccadf79078fc1083927257a18f0e80ff1441ac04eabeff24f97e923e: Status 404 returned error can't find the container with id e05b38edccadf79078fc1083927257a18f0e80ff1441ac04eabeff24f97e923e
Nov 01 20:01:03.096542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:03.096511    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:03.131621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:03.131584    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" event={"ID":"43c131fc-30cf-4216-844f-18c779b82926","Type":"ContainerStarted","Data":"96dd2bfaf87ae95e44bd8b5564b77780451c6797ccd63bed87096e58a56a41b4"}
Nov 01 20:01:03.133172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:03.133141    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" event={"ID":"72595b16-b70f-447e-bba2-1043475ffd59","Type":"ContainerStarted","Data":"e05b38edccadf79078fc1083927257a18f0e80ff1441ac04eabeff24f97e923e"}
Nov 01 20:01:06.159855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:06.159824    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:01:06.160179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:06.159891    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:01:06.582299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:06.582263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:06.681100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:06.681062    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:07.151060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.151027    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" event={"ID":"72595b16-b70f-447e-bba2-1043475ffd59","Type":"ContainerStarted","Data":"9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e"}
Nov 01 20:01:07.151280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.151250    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:07.153123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.153095    6479 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-hgh74 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:01:07.153231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.153156    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" podUID="72595b16-b70f-447e-bba2-1043475ffd59" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:01:07.176263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.176218    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" podStartSLOduration=1.685210222 podStartE2EDuration="5.176205603s" totalImagesPullingTime="3.490995381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:07.17497475 +0000 UTC m=+112.590607233" watchObservedRunningTime="2026-11-01 20:01:07.176205603 +0000 UTC m=+112.591838076"
Nov 01 20:01:07.363428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.363362    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:07.477114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.477059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:07.565899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:07.565863    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:08.160294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:08.160256    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:01:09.162731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:09.162696    6479 generic.go:356] "Generic (PLEG): container finished" podID="43c131fc-30cf-4216-844f-18c779b82926" containerID="68d37225763c92dc80e4cf204d65587e86ed32826c8374f7a2509910c1ce46cf" exitCode=0
Nov 01 20:01:09.163149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:09.162780    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" event={"ID":"43c131fc-30cf-4216-844f-18c779b82926","Type":"ContainerDied","Data":"68d37225763c92dc80e4cf204d65587e86ed32826c8374f7a2509910c1ce46cf"}
Nov 01 20:01:10.019788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:10.019732    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:10.168197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:10.168124    6479 generic.go:356] "Generic (PLEG): container finished" podID="43c131fc-30cf-4216-844f-18c779b82926" containerID="82eb0898f9c673f3ed974dc5e798309fcbf298e41cc058f3bc24a64580d4880a" exitCode=0
Nov 01 20:01:10.168197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:10.168165    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" event={"ID":"43c131fc-30cf-4216-844f-18c779b82926","Type":"ContainerDied","Data":"82eb0898f9c673f3ed974dc5e798309fcbf298e41cc058f3bc24a64580d4880a"}
Nov 01 20:01:16.192887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:16.192856    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" event={"ID":"43c131fc-30cf-4216-844f-18c779b82926","Type":"ContainerStarted","Data":"ae5f025f2a5c3bc5ea3fecac9a6487a4b5490c7349e18f39266eb759e7c73f4b"}
Nov 01 20:01:16.209505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:16.209446    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" podStartSLOduration=2.7007487169999997 podStartE2EDuration="15.209429094s" totalImagesPullingTime="12.508680377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:16.208941564 +0000 UTC m=+121.624574037" watchObservedRunningTime="2026-11-01 20:01:16.209429094 +0000 UTC m=+121.625061570"
Nov 01 20:01:17.198012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:17.197974    6479 generic.go:356] "Generic (PLEG): container finished" podID="43c131fc-30cf-4216-844f-18c779b82926" containerID="ae5f025f2a5c3bc5ea3fecac9a6487a4b5490c7349e18f39266eb759e7c73f4b" exitCode=0
Nov 01 20:01:17.198462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:17.198042    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" event={"ID":"43c131fc-30cf-4216-844f-18c779b82926","Type":"ContainerDied","Data":"ae5f025f2a5c3bc5ea3fecac9a6487a4b5490c7349e18f39266eb759e7c73f4b"}
Nov 01 20:01:18.321999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.321967    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:18.456055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.455970    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util\") pod \"43c131fc-30cf-4216-844f-18c779b82926\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") "
Nov 01 20:01:18.456227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.456076    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle\") pod \"43c131fc-30cf-4216-844f-18c779b82926\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") "
Nov 01 20:01:18.456227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.456109    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh\" (UniqueName: \"kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh\") pod \"43c131fc-30cf-4216-844f-18c779b82926\" (UID: \"43c131fc-30cf-4216-844f-18c779b82926\") "
Nov 01 20:01:18.456604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.456568    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle" pod "43c131fc-30cf-4216-844f-18c779b82926" (UID: "43c131fc-30cf-4216-844f-18c779b82926"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:18.458309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.458283    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh" pod "43c131fc-30cf-4216-844f-18c779b82926" (UID: "43c131fc-30cf-4216-844f-18c779b82926"). InnerVolumeSpecName "kube-api-access-9tjfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:01:18.460147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.460127    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util" pod "43c131fc-30cf-4216-844f-18c779b82926" (UID: "43c131fc-30cf-4216-844f-18c779b82926"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:18.557342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.557304    6479 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-bundle\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:18.557342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.557333    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjfh\" (UniqueName: \"kubernetes.io/projected/43c131fc-30cf-4216-844f-18c779b82926-kube-api-access-9tjfh\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:18.557342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:18.557346    6479 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c131fc-30cf-4216-844f-18c779b82926-util\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:19.206495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:19.206460    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd" event={"ID":"43c131fc-30cf-4216-844f-18c779b82926","Type":"ContainerDied","Data":"96dd2bfaf87ae95e44bd8b5564b77780451c6797ccd63bed87096e58a56a41b4"}
Nov 01 20:01:19.206671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:19.206506    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717xqzd"
Nov 01 20:01:19.206671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:19.206512    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96dd2bfaf87ae95e44bd8b5564b77780451c6797ccd63bed87096e58a56a41b4"
Nov 01 20:01:21.082724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:21.082684    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:21.182609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:21.182571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:29.167787 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:01:29.167747    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:01:33.776438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:33.776400    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:34.672002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.671942    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"]
Nov 01 20:01:34.672420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672402    6479 cpu_manager.go:414] "containerMap: removing container" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="util"
Nov 01 20:01:34.672487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672422    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="util"
Nov 01 20:01:34.672487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672454    6479 cpu_manager.go:414] "containerMap: removing container" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="pull"
Nov 01 20:01:34.672487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672459    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="pull"
Nov 01 20:01:34.672487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672469    6479 cpu_manager.go:414] "containerMap: removing container" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="extract"
Nov 01 20:01:34.672617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672492    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="extract"
Nov 01 20:01:34.672617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.672559    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c131fc-30cf-4216-844f-18c779b82926" containerName="extract"
Nov 01 20:01:34.674685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.674666    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.706681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.706060    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"]
Nov 01 20:01:34.739914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.739884    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 20:01:34.740061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.739978    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 20:01:34.745985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.745919    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 20:01:34.745985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.745983    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 20:01:34.746222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.746062    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 20:01:34.746222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.746132    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rw9ss\""
Nov 01 20:01:34.791746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.791714    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.792125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.791757    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.792125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.791782    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c449c76-8333-45b4-9830-c0147dcd8d35-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.792125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.791895    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c449c76-8333-45b4-9830-c0147dcd8d35-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.792125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.791935    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.792125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.791977    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckln\" (UniqueName: \"kubernetes.io/projected/5c449c76-8333-45b4-9830-c0147dcd8d35-kube-api-access-hckln\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.893248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893213    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.893435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893291    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.893435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893317    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c449c76-8333-45b4-9830-c0147dcd8d35-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.893435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893345    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c449c76-8333-45b4-9830-c0147dcd8d35-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.893435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893370    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.893669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893434    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckln\" (UniqueName: \"kubernetes.io/projected/5c449c76-8333-45b4-9830-c0147dcd8d35-kube-api-access-hckln\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.894002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.893968    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c449c76-8333-45b4-9830-c0147dcd8d35-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.895803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.895772    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c449c76-8333-45b4-9830-c0147dcd8d35-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.895910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.895779    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.895910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.895893    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.895910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.895900    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c449c76-8333-45b4-9830-c0147dcd8d35-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.901002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.900982    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckln\" (UniqueName: \"kubernetes.io/projected/5c449c76-8333-45b4-9830-c0147dcd8d35-kube-api-access-hckln\") pod \"opendatahub-operator-controller-manager-689bcf9f9-kjj7f\" (UID: \"5c449c76-8333-45b4-9830-c0147dcd8d35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:34.927305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.927250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:34.988295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:34.988266    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:35.049560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:35.049530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:35.160287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:35.160256    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:35.281667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:35.281619    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"]
Nov 01 20:01:35.286165 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:01:35.286142    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c449c76_8333_45b4_9830_c0147dcd8d35.slice/crio-3afc6f9845297559222d9d070dd085eaddab3cbcfb1ae8b14a6942a697b7cdbc WatchSource:0}: Error finding container 3afc6f9845297559222d9d070dd085eaddab3cbcfb1ae8b14a6942a697b7cdbc: Status 404 returned error can't find the container with id 3afc6f9845297559222d9d070dd085eaddab3cbcfb1ae8b14a6942a697b7cdbc
Nov 01 20:01:36.266518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:36.266487    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f" event={"ID":"5c449c76-8333-45b4-9830-c0147dcd8d35","Type":"ContainerStarted","Data":"3afc6f9845297559222d9d070dd085eaddab3cbcfb1ae8b14a6942a697b7cdbc"}
Nov 01 20:01:37.199611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:37.199567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:37.313642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:37.313588    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:39.279097 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:39.279060    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f" event={"ID":"5c449c76-8333-45b4-9830-c0147dcd8d35","Type":"ContainerStarted","Data":"a6283665f16d0f566c7c5f497a36a91aa38470ab415fee04380d12424bdb6721"}
Nov 01 20:01:40.287936 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:40.287895    6479 generic.go:356] "Generic (PLEG): container finished" podID="5c449c76-8333-45b4-9830-c0147dcd8d35" containerID="a6283665f16d0f566c7c5f497a36a91aa38470ab415fee04380d12424bdb6721" exitCode=0
Nov 01 20:01:40.288344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:40.287956    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f" event={"ID":"5c449c76-8333-45b4-9830-c0147dcd8d35","Type":"ContainerDied","Data":"a6283665f16d0f566c7c5f497a36a91aa38470ab415fee04380d12424bdb6721"}
Nov 01 20:01:41.136750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:41.136669    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:41.237093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:41.237050    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:41.295324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:41.295289    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f" event={"ID":"5c449c76-8333-45b4-9830-c0147dcd8d35","Type":"ContainerStarted","Data":"32a691af3bbb2be0358a62f05b0807804a712fed2a3d47261671d64276009ce9"}
Nov 01 20:01:41.295749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:41.295408    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:41.331112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:41.331053    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f" podStartSLOduration=2.5141671949999997 podStartE2EDuration="7.331039047s" totalImagesPullingTime="4.340658845s" totalInitContainerRuntime="476.213007ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:41.330449521 +0000 UTC m=+146.746081995" watchObservedRunningTime="2026-11-01 20:01:41.331039047 +0000 UTC m=+146.746671531"
Nov 01 20:01:45.117095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:45.117059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:46.116793 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:46.116755    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:47.332746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:47.332711    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:48.313857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:48.313825    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:52.301551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:52.301523    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-kjj7f"
Nov 01 20:01:53.956266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:53.956229    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:54.106513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:54.106481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:54.211082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:54.210999    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:54.936921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:54.936886    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:55.302105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:55.302061    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:55.441182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:55.441146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:57.248400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:57.248360    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:57.344930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:57.344885    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:58.489858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:58.489819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:59.144685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:59.144636    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:59.168825 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:01:59.168786    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:01:59.249242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:59.249208    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:59.630198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:59.630166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:59.789523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:59.789487    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:01:59.944179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:01:59.944152    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:00.486779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:00.486747    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:02.153079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.153029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:02.262517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.262482    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:02.591130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.591095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:02.618095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.617934    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"]
Nov 01 20:02:02.623414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.623386    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.633542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.633492    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 20:02:02.633542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.633491    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 20:02:02.635000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.634958    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jjbpg\""
Nov 01 20:02:02.636222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.636194    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"]
Nov 01 20:02:02.706083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.706047    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-tls-certs\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.706270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.706167    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-tmp\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.706270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.706240    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9mh\" (UniqueName: \"kubernetes.io/projected/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-kube-api-access-bh9mh\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.799662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.799625    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:02.806850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.806825    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-tls-certs\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.806981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.806872    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-tmp\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.806981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.806894    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9mh\" (UniqueName: \"kubernetes.io/projected/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-kube-api-access-bh9mh\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.809135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.809112    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-tmp\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.809266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.809248    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-tls-certs\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.817704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.817684    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9mh\" (UniqueName: \"kubernetes.io/projected/ff9972fd-5487-4ab7-b0dd-0962e84e88c9-kube-api-access-bh9mh\") pod \"kube-auth-proxy-59c66b786f-wwjm6\" (UID: \"ff9972fd-5487-4ab7-b0dd-0962e84e88c9\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:02.911177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.911100    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:02.940261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:02.940233    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"
Nov 01 20:02:03.015728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:03.015698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:03.157908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:03.157869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:03.210248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:03.210211    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6"]
Nov 01 20:02:03.213278 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:02:03.213246    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9972fd_5487_4ab7_b0dd_0962e84e88c9.slice/crio-aec4d85c23f2ce4616b8831044f0eeaee7ab89970a8dc216ace301072fc95f9b WatchSource:0}: Error finding container aec4d85c23f2ce4616b8831044f0eeaee7ab89970a8dc216ace301072fc95f9b: Status 404 returned error can't find the container with id aec4d85c23f2ce4616b8831044f0eeaee7ab89970a8dc216ace301072fc95f9b
Nov 01 20:02:03.373421 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:03.373386    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6" event={"ID":"ff9972fd-5487-4ab7-b0dd-0962e84e88c9","Type":"ContainerStarted","Data":"aec4d85c23f2ce4616b8831044f0eeaee7ab89970a8dc216ace301072fc95f9b"}
Nov 01 20:02:04.103520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.103484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:04.497356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.497322    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"]
Nov 01 20:02:04.501061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.501032    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.506713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.506606    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 20:02:04.506713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.506615    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-h2wpj\""
Nov 01 20:02:04.506880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.506711    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 20:02:04.510348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.510320    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 20:02:04.511443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.511372    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"]
Nov 01 20:02:04.521023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.520995    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff10696-0cfb-4433-ad91-3d0e817fd274-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.521147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.521056    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eff10696-0cfb-4433-ad91-3d0e817fd274-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.521213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.521187    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eff10696-0cfb-4433-ad91-3d0e817fd274-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.521319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.521245    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjmh\" (UniqueName: \"kubernetes.io/projected/eff10696-0cfb-4433-ad91-3d0e817fd274-kube-api-access-spjmh\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.622556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.622007    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff10696-0cfb-4433-ad91-3d0e817fd274-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.622556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.622070    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eff10696-0cfb-4433-ad91-3d0e817fd274-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.622556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.622141    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eff10696-0cfb-4433-ad91-3d0e817fd274-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.622556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.622187    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjmh\" (UniqueName: \"kubernetes.io/projected/eff10696-0cfb-4433-ad91-3d0e817fd274-kube-api-access-spjmh\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.622937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.622911    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eff10696-0cfb-4433-ad91-3d0e817fd274-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.622997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.622968    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff10696-0cfb-4433-ad91-3d0e817fd274-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.624281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.623525    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eff10696-0cfb-4433-ad91-3d0e817fd274-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.625465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.625437    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:04.637506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.637481    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjmh\" (UniqueName: \"kubernetes.io/projected/eff10696-0cfb-4433-ad91-3d0e817fd274-kube-api-access-spjmh\") pod \"kserve-module-controller-manager-7566cf5fc8-trh22\" (UID: \"eff10696-0cfb-4433-ad91-3d0e817fd274\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:04.819860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:04.819787    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:05.142015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:05.141933    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"]
Nov 01 20:02:05.146233 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:02:05.146199    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff10696_0cfb_4433_ad91_3d0e817fd274.slice/crio-65b56d82b803b36c716309d1af3b3133e85acecb9b12a9ee67ccac5329168b66 WatchSource:0}: Error finding container 65b56d82b803b36c716309d1af3b3133e85acecb9b12a9ee67ccac5329168b66: Status 404 returned error can't find the container with id 65b56d82b803b36c716309d1af3b3133e85acecb9b12a9ee67ccac5329168b66
Nov 01 20:02:05.384455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:05.384420    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22" event={"ID":"eff10696-0cfb-4433-ad91-3d0e817fd274","Type":"ContainerStarted","Data":"65b56d82b803b36c716309d1af3b3133e85acecb9b12a9ee67ccac5329168b66"}
Nov 01 20:02:05.954450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:05.954403    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:06.051669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:06.051607    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:06.366853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:06.366817    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:06.390025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:06.389987    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6" event={"ID":"ff9972fd-5487-4ab7-b0dd-0962e84e88c9","Type":"ContainerStarted","Data":"582268a4f284a593e51f9fd32e58c5fd04c84087ec17d66c945dcbe3a4b5b613"}
Nov 01 20:02:06.414441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:06.414394    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59c66b786f-wwjm6" podStartSLOduration=1.810634831 podStartE2EDuration="4.414380747s" totalImagesPullingTime="2.603745916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:06.414062559 +0000 UTC m=+171.829695032" watchObservedRunningTime="2026-11-01 20:02:06.414380747 +0000 UTC m=+171.830013217"
Nov 01 20:02:07.211309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:07.211195    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:07.395995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:07.395957    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22" event={"ID":"eff10696-0cfb-4433-ad91-3d0e817fd274","Type":"ContainerStarted","Data":"b8ffd8a8639e5d9716db57fc6bbeafa56997143e85038816fb5ebfca5aaabf7f"}
Nov 01 20:02:07.419324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:07.419086    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22" podStartSLOduration=1.6207143670000002 podStartE2EDuration="3.419068422s" totalImagesPullingTime="1.798354055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:07.418359424 +0000 UTC m=+172.833991910" watchObservedRunningTime="2026-11-01 20:02:07.419068422 +0000 UTC m=+172.834700896"
Nov 01 20:02:08.399818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:08.399783    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:08.752723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:08.752689    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:10.125519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:10.125481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:10.247372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:10.247335    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:10.720140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:10.720083    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:10.820628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:10.820595    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:11.323609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:11.323574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:11.402154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:11.402122    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:16.738112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.738079    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"]
Nov 01 20:02:16.743790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.743769    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:16.753446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.753418    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.754281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.754257    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.754632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.754613    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-swjbx\""
Nov 01 20:02:16.755934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.755909    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"]
Nov 01 20:02:16.823530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.823494    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhpm\" (UniqueName: \"kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm\") pod \"kuadrant-operator-controller-manager-678d87766f-59thg\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:16.823719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.823538    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-59thg\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:16.924564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.924523    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhpm\" (UniqueName: \"kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm\") pod \"kuadrant-operator-controller-manager-678d87766f-59thg\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:16.924766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.924580    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-59thg\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:16.925017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.924996    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-59thg\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:16.932785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:16.932767    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhpm\" (UniqueName: \"kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm\") pod \"kuadrant-operator-controller-manager-678d87766f-59thg\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:17.055526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:17.055451    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:17.327953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:17.326980    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"]
Nov 01 20:02:17.329921 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:02:17.329895    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f62c32d_0014_4bf3_853c_dd0cfc0f6b66.slice/crio-25c30c102954e330c68304fefe64c167f9226716bc334d25e8ff9f31c2b01afa WatchSource:0}: Error finding container 25c30c102954e330c68304fefe64c167f9226716bc334d25e8ff9f31c2b01afa: Status 404 returned error can't find the container with id 25c30c102954e330c68304fefe64c167f9226716bc334d25e8ff9f31c2b01afa
Nov 01 20:02:17.434502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:17.434465    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg" event={"ID":"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66","Type":"ContainerStarted","Data":"25c30c102954e330c68304fefe64c167f9226716bc334d25e8ff9f31c2b01afa"}
Nov 01 20:02:19.131507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:19.131450    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:19.234590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:19.234553    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:19.339195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:19.339157    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:19.433074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:19.432987    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:21.135089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:21.135050    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:21.369043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:21.368999    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:21.449902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:21.449869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:22.458208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:22.458174    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg" event={"ID":"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66","Type":"ContainerStarted","Data":"70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164"}
Nov 01 20:02:22.458599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:22.458248    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:22.479071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:22.479017    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg" podStartSLOduration=2.090192814 podStartE2EDuration="6.478999903s" totalImagesPullingTime="4.388807089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:22.478426785 +0000 UTC m=+187.894059271" watchObservedRunningTime="2026-11-01 20:02:22.478999903 +0000 UTC m=+187.894632374"
Nov 01 20:02:26.126705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:26.126607    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:26.228205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:26.228143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:27.115050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:27.115016    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:27.187041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:27.187010    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:29.169444 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:02:29.169407    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:02:29.404356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:29.404327    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-trh22"
Nov 01 20:02:30.952634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:30.952600    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:31.051503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:31.051467    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:33.464913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:33.464881    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:02:39.768016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:39.767964    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:39.868980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:39.868950    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:42.225492 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:42.225441    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:42.295680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:42.295629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:45.338261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:45.338231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:46.969463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:46.969427    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:47.069063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:47.069024    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:48.288167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:48.288131    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:48.355849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:48.355814    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:51.513087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.513054    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:51.611573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.611532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:51.688968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.688934    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:51.775808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.775718    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:51.884840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.884809    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:02:51.889912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.889890    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:51.900541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.900515    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4pgmt\""
Nov 01 20:02:51.900700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.900557    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 20:02:51.904422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.904380    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:02:51.941237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.941202    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c6137ad-fb29-4667-be40-173018e00d94-config-file\") pod \"limitador-limitador-7c9b87d574-v5k66\" (UID: \"8c6137ad-fb29-4667-be40-173018e00d94\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:51.941422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.941273    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c92t\" (UniqueName: \"kubernetes.io/projected/8c6137ad-fb29-4667-be40-173018e00d94-kube-api-access-9c92t\") pod \"limitador-limitador-7c9b87d574-v5k66\" (UID: \"8c6137ad-fb29-4667-be40-173018e00d94\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:51.998994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:51.998962    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:02:52.042670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.042573    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c92t\" (UniqueName: \"kubernetes.io/projected/8c6137ad-fb29-4667-be40-173018e00d94-kube-api-access-9c92t\") pod \"limitador-limitador-7c9b87d574-v5k66\" (UID: \"8c6137ad-fb29-4667-be40-173018e00d94\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:52.042816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.042672    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c6137ad-fb29-4667-be40-173018e00d94-config-file\") pod \"limitador-limitador-7c9b87d574-v5k66\" (UID: \"8c6137ad-fb29-4667-be40-173018e00d94\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:52.043367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.043349    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c6137ad-fb29-4667-be40-173018e00d94-config-file\") pod \"limitador-limitador-7c9b87d574-v5k66\" (UID: \"8c6137ad-fb29-4667-be40-173018e00d94\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:52.054416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.054384    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c92t\" (UniqueName: \"kubernetes.io/projected/8c6137ad-fb29-4667-be40-173018e00d94-kube-api-access-9c92t\") pod \"limitador-limitador-7c9b87d574-v5k66\" (UID: \"8c6137ad-fb29-4667-be40-173018e00d94\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:52.200550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.200517    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:52.201431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.201414    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:52.288749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.288715    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:52.451182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.451146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:52.499743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.499712    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:02:52.502993 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:02:52.502959    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6137ad_fb29_4667_be40_173018e00d94.slice/crio-28718a0e4d49aeae5d709cdca4373506abdd486bc4147332d03dd67de0996086 WatchSource:0}: Error finding container 28718a0e4d49aeae5d709cdca4373506abdd486bc4147332d03dd67de0996086: Status 404 returned error can't find the container with id 28718a0e4d49aeae5d709cdca4373506abdd486bc4147332d03dd67de0996086
Nov 01 20:02:52.582069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.582031    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66" event={"ID":"8c6137ad-fb29-4667-be40-173018e00d94","Type":"ContainerStarted","Data":"28718a0e4d49aeae5d709cdca4373506abdd486bc4147332d03dd67de0996086"}
Nov 01 20:02:52.653925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.653890    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:52.761597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.761518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:52.872052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:52.872022    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:54.591942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:54.591905    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66" event={"ID":"8c6137ad-fb29-4667-be40-173018e00d94","Type":"ContainerStarted","Data":"c691afd744c490f130d2bc17bfdcd4d1c96ff0f62fb052d33e86bb4caa037b92"}
Nov 01 20:02:54.592364 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:54.592019    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:02:54.618016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:54.617955    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66" podStartSLOduration=2.441308961 podStartE2EDuration="3.617940078s" totalImagesPullingTime="1.176631117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:54.615919074 +0000 UTC m=+220.031551560" watchObservedRunningTime="2026-11-01 20:02:54.617940078 +0000 UTC m=+220.033572552"
Nov 01 20:02:56.564917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.564884    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:56.672147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.672115    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:56.774159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.774123    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"]
Nov 01 20:02:56.776353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.776335    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.777171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.777147    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:56.788125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.788101    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"]
Nov 01 20:02:56.874921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.874830    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:56.887971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.887936    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b32e8-0368-46cc-9385-ba469e82c957-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.887971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.887975    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60b32e8-0368-46cc-9385-ba469e82c957-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.888227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.888001    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b32e8-0368-46cc-9385-ba469e82c957-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.888227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.888115    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.888227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.888158    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.888326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.888245    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.888326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.888279    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fz4\" (UniqueName: \"kubernetes.io/projected/c60b32e8-0368-46cc-9385-ba469e82c957-kube-api-access-d5fz4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989405    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989457    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989476    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fz4\" (UniqueName: \"kubernetes.io/projected/c60b32e8-0368-46cc-9385-ba469e82c957-kube-api-access-d5fz4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989516    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b32e8-0368-46cc-9385-ba469e82c957-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989534    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60b32e8-0368-46cc-9385-ba469e82c957-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989555    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b32e8-0368-46cc-9385-ba469e82c957-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.989639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.989601    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.990444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.990411    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60b32e8-0368-46cc-9385-ba469e82c957-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.991989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.991962    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.992087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.991962    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b32e8-0368-46cc-9385-ba469e82c957-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.992134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.992116    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:56.992215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:56.992196    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60b32e8-0368-46cc-9385-ba469e82c957-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:57.010381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.010350    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b32e8-0368-46cc-9385-ba469e82c957-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:57.010539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.010513    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fz4\" (UniqueName: \"kubernetes.io/projected/c60b32e8-0368-46cc-9385-ba469e82c957-kube-api-access-d5fz4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k44gg\" (UID: \"c60b32e8-0368-46cc-9385-ba469e82c957\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:57.053888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.053852    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:57.087436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.087404    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:57.368974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.368929    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"]
Nov 01 20:02:57.373060 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:02:57.373026    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60b32e8_0368_46cc_9385_ba469e82c957.slice/crio-78e0b25ae45b452994f10012f411e4898e6563468afef803103df7e60e5a90e8 WatchSource:0}: Error finding container 78e0b25ae45b452994f10012f411e4898e6563468afef803103df7e60e5a90e8: Status 404 returned error can't find the container with id 78e0b25ae45b452994f10012f411e4898e6563468afef803103df7e60e5a90e8
Nov 01 20:02:57.375317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.375284    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:02:57.375405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.375346    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:02:57.608373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.608333    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg" event={"ID":"c60b32e8-0368-46cc-9385-ba469e82c957","Type":"ContainerStarted","Data":"04c0e450d3cfc58130592e4a99b52e9ad09fcf5cf1fb53218048bcafc8a37205"}
Nov 01 20:02:57.608864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.608410    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg" event={"ID":"c60b32e8-0368-46cc-9385-ba469e82c957","Type":"ContainerStarted","Data":"78e0b25ae45b452994f10012f411e4898e6563468afef803103df7e60e5a90e8"}
Nov 01 20:02:57.608864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.608447    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:57.629855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.629795    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg" podStartSLOduration=1.62978066 podStartE2EDuration="1.62978066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:57.628437807 +0000 UTC m=+223.044070284" watchObservedRunningTime="2026-11-01 20:02:57.62978066 +0000 UTC m=+223.045413132"
Nov 01 20:02:57.675567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.675474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:57.788841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.788800    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:57.939063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:57.938974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:02:58.615099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:58.615067    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k44gg"
Nov 01 20:02:58.667188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:58.667152    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"]
Nov 01 20:02:58.667416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:58.667392    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" podUID="72595b16-b70f-447e-bba2-1043475ffd59" containerName="discovery" containerID="cri-o://9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e" gracePeriod=30
Nov 01 20:02:58.910406 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:58.910378    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:02:59.011811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011778    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.011811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011819    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.012064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011857    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.012064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011886    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.012064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011923    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.012064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011949    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.012064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.011977    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig\") pod \"72595b16-b70f-447e-bba2-1043475ffd59\" (UID: \"72595b16-b70f-447e-bba2-1043475ffd59\") "
Nov 01 20:02:59.012334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.012282    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:02:59.014322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.014286    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:02:59.014443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.014325    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "kube-api-access-rwm2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:02:59.014518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.014499    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:02:59.014572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.014531    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:02:59.014608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.014591    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:02:59.014641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.014608    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert" pod "72595b16-b70f-447e-bba2-1043475ffd59" (UID: "72595b16-b70f-447e-bba2-1043475ffd59"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:02:59.113218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113174    6479 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-dns-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.113218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113211    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm2h\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-kube-api-access-rwm2h\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.113218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113223    6479 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-istio-kubeconfig\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.113477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113233    6479 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72595b16-b70f-447e-bba2-1043475ffd59-cacerts\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.113477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113243    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72595b16-b70f-447e-bba2-1043475ffd59-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.113477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113251    6479 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72595b16-b70f-447e-bba2-1043475ffd59-istio-csr-ca-configmap\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.113477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.113261    6479 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72595b16-b70f-447e-bba2-1043475ffd59-local-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.169970 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:02:59.169891    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:02:59.618573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.618539    6479 generic.go:356] "Generic (PLEG): container finished" podID="72595b16-b70f-447e-bba2-1043475ffd59" containerID="9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e" exitCode=0
Nov 01 20:02:59.619060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.618593    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"
Nov 01 20:02:59.619060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.618636    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" event={"ID":"72595b16-b70f-447e-bba2-1043475ffd59","Type":"ContainerDied","Data":"9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e"}
Nov 01 20:02:59.619060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.618712    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74" event={"ID":"72595b16-b70f-447e-bba2-1043475ffd59","Type":"ContainerDied","Data":"e05b38edccadf79078fc1083927257a18f0e80ff1441ac04eabeff24f97e923e"}
Nov 01 20:02:59.619060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.618770    6479 scope.go:122] "RemoveContainer" containerID="9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e"
Nov 01 20:02:59.630092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.630068    6479 scope.go:122] "RemoveContainer" containerID="9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e"
Nov 01 20:02:59.630401 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:02:59.630378    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e\": container with ID starting with 9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e not found: ID does not exist" containerID="9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e"
Nov 01 20:02:59.630444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.630412    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e"} err="failed to get container status \"9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e\": rpc error: code = NotFound desc = could not find container \"9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e\": container with ID starting with 9e237589ff731ea12ceddc9e177c9267a2474fdee15294905462d3af8a64ca5e not found: ID does not exist"
Nov 01 20:02:59.640275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.640248    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"]
Nov 01 20:02:59.644743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:02:59.644716    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgh74"]
Nov 01 20:03:00.705505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:00.705472    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72595b16-b70f-447e-bba2-1043475ffd59" path="/var/lib/kubelet/pods/72595b16-b70f-447e-bba2-1043475ffd59/volumes"
Nov 01 20:03:01.185529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:01.185454    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:01.285242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:01.285210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:03.636399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:03.636361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:03.737396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:03.737361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:05.599160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:05.599129    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v5k66"
Nov 01 20:03:08.381468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:08.381430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:08.481574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:08.481525    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:10.945840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:10.945808    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:11.048102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:11.048069    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:11.692229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:11.692198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:21.157561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:21.157525    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:29.169212 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:03:29.169172    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:03:30.425612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:30.425578    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:30.507101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:30.507064    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:30.581406 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:30.581376    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:30.682958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:30.682866    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:34.360538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:34.360505    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:34.437540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:34.437507    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:35.325237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:35.325197    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:35.425602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:35.425566    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:36.785158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:36.785124    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:36.886012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:03:36.885974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:59.169901 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:03:59.169817    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:04:06.159018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:06.158970    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:06.254349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:06.254319    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:07.446352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:07.446315    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:07.549498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:07.549459    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:07.778864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:07.778778    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:08.259754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:08.259722    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:04:19.410893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.410858    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:19.571423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.571387    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:19.881581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.881548    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"]
Nov 01 20:04:19.881909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.881896    6479 cpu_manager.go:414] "containerMap: removing container" podUID="72595b16-b70f-447e-bba2-1043475ffd59" containerName="discovery"
Nov 01 20:04:19.881958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.881910    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72595b16-b70f-447e-bba2-1043475ffd59" containerName="discovery"
Nov 01 20:04:19.881992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.881968    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="72595b16-b70f-447e-bba2-1043475ffd59" containerName="discovery"
Nov 01 20:04:19.884963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.884943    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:19.891571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.891157    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 20:04:19.892768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.892522    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7vx47\""
Nov 01 20:04:19.892768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.892611    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 20:04:19.893092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.893073    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:04:19.893170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.893096    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:04:19.894266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.894245    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"]
Nov 01 20:04:19.902491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.902467    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:19.902594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.902523    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:19.902667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.902591    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2xn\" (UniqueName: \"kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:19.902667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:19.902655    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.003077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.003045    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.003264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.003120    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.003264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.003146    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2xn\" (UniqueName: \"kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.003264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.003170    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.003930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.003896    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.005702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.005677    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.005764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.005683    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.013856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.013830    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2xn\" (UniqueName: \"kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn\") pod \"maas-api-8f9dc8cf-qlxbq\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.197956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.197922    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:20.472238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.472156    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"]
Nov 01 20:04:20.475333 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:04:20.475303    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a63c17_402b_4604_bf9c_e4184b62c283.slice/crio-edf54fe8175a95ec6882d84193b614ee36184cae29a45deae674fcf6d056c767 WatchSource:0}: Error finding container edf54fe8175a95ec6882d84193b614ee36184cae29a45deae674fcf6d056c767: Status 404 returned error can't find the container with id edf54fe8175a95ec6882d84193b614ee36184cae29a45deae674fcf6d056c767
Nov 01 20:04:20.963188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:20.963151    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" event={"ID":"58a63c17-402b-4604-bf9c-e4184b62c283","Type":"ContainerStarted","Data":"edf54fe8175a95ec6882d84193b614ee36184cae29a45deae674fcf6d056c767"}
Nov 01 20:04:21.208138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:21.208098    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:22.979286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:22.979243    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" event={"ID":"58a63c17-402b-4604-bf9c-e4184b62c283","Type":"ContainerStarted","Data":"f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792"}
Nov 01 20:04:22.979755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:22.979318    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:22.996413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:22.996363    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" podStartSLOduration=1.645128318 podStartE2EDuration="3.996350571s" totalImagesPullingTime="2.351222253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:04:22.994748327 +0000 UTC m=+308.410380799" watchObservedRunningTime="2026-11-01 20:04:22.996350571 +0000 UTC m=+308.411983044"
Nov 01 20:04:28.988727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:28.988694    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:04:29.169517 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:04:29.169482    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:04:58.169943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:58.169905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:58.267723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:04:58.267690    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:04:59.168565 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:04:59.168527    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:05:03.258819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:03.258784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:03.352748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:03.352719    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:11.419400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:11.419332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:11.519870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:11.519820    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:21.246582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:21.246545    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:28.236949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:28.236870    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:28.722161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:28.722124    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:29.168677 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:05:29.168578    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:05:29.874855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:29.874821    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:29.975712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:29.975674    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:32.635687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:32.635629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:33.042040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:33.042004    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:05:35.292888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:35.292854    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:35.638403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:35.638325    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:05:36.567182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.567139    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"]
Nov 01 20:05:36.571135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.571114    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.575909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.575885    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ww25v\""
Nov 01 20:05:36.578902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.578878    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 20:05:36.580171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.580151    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 20:05:36.580631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.580606    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 20:05:36.581346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.581328    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"]
Nov 01 20:05:36.619102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.619070    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:36.634464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.634422    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.634591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.634474    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcxr\" (UniqueName: \"kubernetes.io/projected/159ab949-f5cd-452a-94ca-89c96023149d-kube-api-access-kpcxr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.634699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.634587    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.634699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.634671    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.634699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.634693    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ab949-f5cd-452a-94ca-89c96023149d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.634829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.634716    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.735835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.735798    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.735835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.735838    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ab949-f5cd-452a-94ca-89c96023149d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.735870    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.735942    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.735976    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcxr\" (UniqueName: \"kubernetes.io/projected/159ab949-f5cd-452a-94ca-89c96023149d-kube-api-access-kpcxr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.736022    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.736369    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.736377    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.736694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.736628    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.738414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.738393    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159ab949-f5cd-452a-94ca-89c96023149d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.738511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.738445    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159ab949-f5cd-452a-94ca-89c96023149d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.743735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.743717    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcxr\" (UniqueName: \"kubernetes.io/projected/159ab949-f5cd-452a-94ca-89c96023149d-kube-api-access-kpcxr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fhmpr\" (UID: \"159ab949-f5cd-452a-94ca-89c96023149d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:36.887326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:36.887246    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:37.132848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:37.132812    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:05:37.162473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:37.162443    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"]
Nov 01 20:05:37.165500 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:05:37.165470    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159ab949_f5cd_452a_94ca_89c96023149d.slice/crio-9e7075542433752152b5cd4e0474afd0cc061bd1e5883ad0493fd87c1a0b5e12 WatchSource:0}: Error finding container 9e7075542433752152b5cd4e0474afd0cc061bd1e5883ad0493fd87c1a0b5e12: Status 404 returned error can't find the container with id 9e7075542433752152b5cd4e0474afd0cc061bd1e5883ad0493fd87c1a0b5e12
Nov 01 20:05:37.290003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:37.289965    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr" event={"ID":"159ab949-f5cd-452a-94ca-89c96023149d","Type":"ContainerStarted","Data":"9e7075542433752152b5cd4e0474afd0cc061bd1e5883ad0493fd87c1a0b5e12"}
Nov 01 20:05:38.522827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:38.522786    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:39.034639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.034605    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:05:39.048581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.048554    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"]
Nov 01 20:05:39.052025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.052000    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.064452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.060428    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 20:05:39.064452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.062026    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"]
Nov 01 20:05:39.116076 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.116048    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:39.171193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.171150    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.171193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.171187    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmd7\" (UniqueName: \"kubernetes.io/projected/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-kube-api-access-qwmd7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.171484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.171214    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.171484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.171294    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.171484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.171325    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.171676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.171582    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.272591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.272552    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.272780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.272642    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.272780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.272710    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.272780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.272736    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmd7\" (UniqueName: \"kubernetes.io/projected/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-kube-api-access-qwmd7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.272780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.272773    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.273000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.272923    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.273151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.273122    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.273151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.273147    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.273327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.273187    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.274890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.274864    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.275172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.275153    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.284174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.284150    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmd7\" (UniqueName: \"kubernetes.io/projected/bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3-kube-api-access-qwmd7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2\" (UID: \"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.300378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.300305    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr" event={"ID":"159ab949-f5cd-452a-94ca-89c96023149d","Type":"ContainerStarted","Data":"76827e7cc610c0334abb22513170ad23289fa44d21c4f74fbdb75c165ef379a6"}
Nov 01 20:05:39.318235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.318183    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr" podStartSLOduration=2.158818473 podStartE2EDuration="3.31816721s" totalImagesPullingTime="1.159348737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:39.315893126 +0000 UTC m=+384.731525611" watchObservedRunningTime="2026-11-01 20:05:39.31816721 +0000 UTC m=+384.733799686"
Nov 01 20:05:39.370998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.370962    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:39.642617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.642498    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"]
Nov 01 20:05:39.646177 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:05:39.646144    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcf6c19_7418_4ed0_ac7c_9abaed9df5b3.slice/crio-a70f0737c9ff623026e79aba898825eed11fd0c25aea8a6fa6cb5daa3bad726c WatchSource:0}: Error finding container a70f0737c9ff623026e79aba898825eed11fd0c25aea8a6fa6cb5daa3bad726c: Status 404 returned error can't find the container with id a70f0737c9ff623026e79aba898825eed11fd0c25aea8a6fa6cb5daa3bad726c
Nov 01 20:05:39.888226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:39.888194    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:40.306121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.306016    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2" event={"ID":"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3","Type":"ContainerStarted","Data":"363f6554d66c3d2886574e31d6d5df7ac7c055b5d2819804a007c621fd93e62b"}
Nov 01 20:05:40.306121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.306085    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2" event={"ID":"bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3","Type":"ContainerStarted","Data":"a70f0737c9ff623026e79aba898825eed11fd0c25aea8a6fa6cb5daa3bad726c"}
Nov 01 20:05:40.323569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.323517    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2" podStartSLOduration=1.02114247 podStartE2EDuration="1.323503939s" totalImagesPullingTime="302.361469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:40.322201033 +0000 UTC m=+385.737833517" watchObservedRunningTime="2026-11-01 20:05:40.323503939 +0000 UTC m=+385.739136411"
Nov 01 20:05:40.371981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.371936    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:40.413297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.412797    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bl4xv"]
Nov 01 20:05:40.416836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.416809    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.424525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.424498    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 20:05:40.425761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.425738    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bl4xv"]
Nov 01 20:05:40.477217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.477189    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:40.484876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.484849    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.484985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.484893    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac54c9bd-b9df-47d3-b566-73a43f6263e5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.484985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.484974    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-home\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.485051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.485033    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.485117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.485102    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr64\" (UniqueName: \"kubernetes.io/projected/ac54c9bd-b9df-47d3-b566-73a43f6263e5-kube-api-access-qrr64\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.485158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.485138    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.585888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.585810    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-home\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.585886    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.585923    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr64\" (UniqueName: \"kubernetes.io/projected/ac54c9bd-b9df-47d3-b566-73a43f6263e5-kube-api-access-qrr64\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.585957    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.586029    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.586054    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac54c9bd-b9df-47d3-b566-73a43f6263e5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.586290    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-home\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.586565    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.586714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.586600    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.588619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.588598    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac54c9bd-b9df-47d3-b566-73a43f6263e5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.588771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.588751    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac54c9bd-b9df-47d3-b566-73a43f6263e5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.593953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.593930    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr64\" (UniqueName: \"kubernetes.io/projected/ac54c9bd-b9df-47d3-b566-73a43f6263e5-kube-api-access-qrr64\") pod \"echo-simulator-kserve-64b6f8f96c-bl4xv\" (UID: \"ac54c9bd-b9df-47d3-b566-73a43f6263e5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:40.737213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:40.737185    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:41.012210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:41.012176    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bl4xv"]
Nov 01 20:05:41.015296 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:05:41.015253    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac54c9bd_b9df_47d3_b566_73a43f6263e5.slice/crio-79e4c7be44b6cafe1f122f716d58f78e822eae68f05055bb694d448aa6ff30f3 WatchSource:0}: Error finding container 79e4c7be44b6cafe1f122f716d58f78e822eae68f05055bb694d448aa6ff30f3: Status 404 returned error can't find the container with id 79e4c7be44b6cafe1f122f716d58f78e822eae68f05055bb694d448aa6ff30f3
Nov 01 20:05:41.103923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:41.103887    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:41.311315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:41.311224    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv" event={"ID":"ac54c9bd-b9df-47d3-b566-73a43f6263e5","Type":"ContainerStarted","Data":"79e4c7be44b6cafe1f122f716d58f78e822eae68f05055bb694d448aa6ff30f3"}
Nov 01 20:05:42.109061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:42.109023    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:05:42.513103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:42.513073    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:43.321561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:43.321520    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv" event={"ID":"ac54c9bd-b9df-47d3-b566-73a43f6263e5","Type":"ContainerStarted","Data":"b7cc4e337c89a9fbd9050f45ccf124af168feb31ffd6db10524755f33cce8351"}
Nov 01 20:05:43.338614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:43.338559    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv" podStartSLOduration=1.5215660149999999 podStartE2EDuration="3.338541873s" totalImagesPullingTime="1.816975858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:43.337948452 +0000 UTC m=+388.753580927" watchObservedRunningTime="2026-11-01 20:05:43.338541873 +0000 UTC m=+388.754174347"
Nov 01 20:05:43.737466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:43.737427    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:46.888012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:46.887971    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:46.900940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:46.900914    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:47.185056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:47.184970    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:05:47.349263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:47.349236    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fhmpr"
Nov 01 20:05:49.371564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:49.371529    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:49.389990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:49.389961    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:50.363355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:50.363326    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2"
Nov 01 20:05:50.737980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:50.737952    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:50.750279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:50.750251    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:51.377461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:51.377429    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bl4xv"
Nov 01 20:05:57.857890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:57.857850    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:57.989347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:57.989309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:59.174795 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:05:59.174754    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:05:59.310295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:59.310259    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:05:59.435484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:05:59.435399    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:07.702484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.702445    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"]
Nov 01 20:06:07.708971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.708943    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.720684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.719344    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"]
Nov 01 20:06:07.743519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743478    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743537    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582rw\" (UniqueName: \"kubernetes.io/projected/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-kube-api-access-582rw\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743621    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743710    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743738    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743819    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743855    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743876    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.743989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.743960    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.805756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.805724    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-njf76\""
Nov 01 20:06:07.844871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.844834    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.844887    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-582rw\" (UniqueName: \"kubernetes.io/projected/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-kube-api-access-582rw\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.844904    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.844943    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.844967    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.844992    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845388    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845449    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845483    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845496    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845508    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845565    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845702    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.845818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.845788    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.847608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.847581    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.847736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.847613    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.853294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.853265    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:07.853407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:07.853357    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-582rw\" (UniqueName: \"kubernetes.io/projected/c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5-kube-api-access-582rw\") pod \"maas-default-gateway-openshift-default-fd4b99879-c6qrt\" (UID: \"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:08.036785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:08.036690    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:08.314277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:08.314251    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"]
Nov 01 20:06:08.317611 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:06:08.317563    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f59fb5_3dd5_4c6b_a1fb_94ffb28812a5.slice/crio-d8d7c6ecbe95243f03b377cc4df63faa4b87dbc88a37a90328fd72156dc9f32f WatchSource:0}: Error finding container d8d7c6ecbe95243f03b377cc4df63faa4b87dbc88a37a90328fd72156dc9f32f: Status 404 returned error can't find the container with id d8d7c6ecbe95243f03b377cc4df63faa4b87dbc88a37a90328fd72156dc9f32f
Nov 01 20:06:08.319434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:08.319403    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:06:08.438587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:08.438549    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" event={"ID":"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5","Type":"ContainerStarted","Data":"d8d7c6ecbe95243f03b377cc4df63faa4b87dbc88a37a90328fd72156dc9f32f"}
Nov 01 20:06:11.462570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:11.462539    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:11.462891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:11.462607    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:11.462891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:11.462637    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:12.483045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:12.483002    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" event={"ID":"c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5","Type":"ContainerStarted","Data":"818deb72ae16e3c029927f4b3fea171ef84eda355ab4d537002d8f20a5de896a"}
Nov 01 20:06:12.501957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:12.501905    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podStartSLOduration=2.360047874 podStartE2EDuration="5.501884803s" totalImagesPullingTime="3.141836929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:12.500299619 +0000 UTC m=+417.915932125" watchObservedRunningTime="2026-11-01 20:06:12.501884803 +0000 UTC m=+417.917517280"
Nov 01 20:06:13.037832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:13.037798    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:13.039192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:13.039164    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Nov 01 20:06:13.039301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:13.039219    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Nov 01 20:06:14.037413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:14.037374    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Nov 01 20:06:14.037820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:14.037442    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Nov 01 20:06:15.037603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:15.037563    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Nov 01 20:06:15.038020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:15.037620    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Nov 01 20:06:16.037624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:16.037584    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Nov 01 20:06:16.038121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:16.037686    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Nov 01 20:06:17.037459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:17.037422    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Nov 01 20:06:17.037632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:17.037503    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Nov 01 20:06:18.037407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:18.037369    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Nov 01 20:06:18.037810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:18.037424    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Nov 01 20:06:19.743310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:19.743274    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:19.743822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:19.743695    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:19.744581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:19.744560    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt"
Nov 01 20:06:21.292232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:21.292196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:22.706731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.706699    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"]
Nov 01 20:06:22.710471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.710454    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.723813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.723783    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"]
Nov 01 20:06:22.727948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.727917    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"]
Nov 01 20:06:22.731144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.731123    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.744226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.744187    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"]
Nov 01 20:06:22.784482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784440    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.784482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784478    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.784771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784495    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.784771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784613    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.784771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784640    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.784771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784677    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.784771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784744    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.784771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784765    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2mr\" (UniqueName: \"kubernetes.io/projected/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-kube-api-access-vq2mr\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784795    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784840    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7wv\" (UniqueName: \"kubernetes.io/projected/838f2059-c8d7-4b2c-95e9-a9a31c87683c-kube-api-access-4g7wv\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784866    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784899    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784978    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.784997    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.785015    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.785095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.785034    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.785571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.785162    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.785571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.785194    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.885903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.885864    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.885903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.885905    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.885925    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.885949    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886010    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886052    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886102    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886124    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886146    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886198    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886224    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886248    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886277    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886303    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2mr\" (UniqueName: \"kubernetes.io/projected/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-kube-api-access-vq2mr\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886330    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886353    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7wv\" (UniqueName: \"kubernetes.io/projected/838f2059-c8d7-4b2c-95e9-a9a31c87683c-kube-api-access-4g7wv\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886377    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886413    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.886451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886418    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886471    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886475    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886517    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886552    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886741    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.886853    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.887037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.887031    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.887391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.887260    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.887391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.887267    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.888967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.888921    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.889098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.889074    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.889143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.889115    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.889331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.889315    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.894789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.894766    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7wv\" (UniqueName: \"kubernetes.io/projected/838f2059-c8d7-4b2c-95e9-a9a31c87683c-kube-api-access-4g7wv\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.894929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.894911    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:22.895147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.895128    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838f2059-c8d7-4b2c-95e9-a9a31c87683c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jsxtq\" (UID: \"838f2059-c8d7-4b2c-95e9-a9a31c87683c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:22.895608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:22.895592    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2mr\" (UniqueName: \"kubernetes.io/projected/020b4fc0-677d-4ec2-ac2d-bdd1379e7382-kube-api-access-vq2mr\") pod \"maas-default-gateway-openshift-default-fd4b99879-s4pct\" (UID: \"020b4fc0-677d-4ec2-ac2d-bdd1379e7382\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:23.028479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.028400    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:23.051924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.051897    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:23.306110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.306082    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"]
Nov 01 20:06:23.308683 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:06:23.308632    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f2059_c8d7_4b2c_95e9_a9a31c87683c.slice/crio-867222c29a8851b6506980219d4f790c5ca2b346ac9efff507d3e0f490fd00d6 WatchSource:0}: Error finding container 867222c29a8851b6506980219d4f790c5ca2b346ac9efff507d3e0f490fd00d6: Status 404 returned error can't find the container with id 867222c29a8851b6506980219d4f790c5ca2b346ac9efff507d3e0f490fd00d6
Nov 01 20:06:23.310855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.310824    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:23.310922 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.310894    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:23.310958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.310927    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:23.335875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.335850    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"]
Nov 01 20:06:23.341912 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:06:23.341872    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020b4fc0_677d_4ec2_ac2d_bdd1379e7382.slice/crio-7b99af1bec0751ea20434d4ea7905fd9b1b9720a7f8488289c505e601216ffcd WatchSource:0}: Error finding container 7b99af1bec0751ea20434d4ea7905fd9b1b9720a7f8488289c505e601216ffcd: Status 404 returned error can't find the container with id 7b99af1bec0751ea20434d4ea7905fd9b1b9720a7f8488289c505e601216ffcd
Nov 01 20:06:23.344961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.344917    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:23.345059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.345020    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:23.345108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.345067    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:23.532098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.532054    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" event={"ID":"838f2059-c8d7-4b2c-95e9-a9a31c87683c","Type":"ContainerStarted","Data":"6142ef315a33161743152e4de4afa73986adc85c6f6263bbaff15c8111bd0e90"}
Nov 01 20:06:23.532275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.532131    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" event={"ID":"838f2059-c8d7-4b2c-95e9-a9a31c87683c","Type":"ContainerStarted","Data":"867222c29a8851b6506980219d4f790c5ca2b346ac9efff507d3e0f490fd00d6"}
Nov 01 20:06:23.534270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.534231    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" event={"ID":"020b4fc0-677d-4ec2-ac2d-bdd1379e7382","Type":"ContainerStarted","Data":"53aed6c2035d2fdf83616aa082982d0edcf90f447a666e30364def51a1d8c0f4"}
Nov 01 20:06:23.534385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.534287    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" event={"ID":"020b4fc0-677d-4ec2-ac2d-bdd1379e7382","Type":"ContainerStarted","Data":"7b99af1bec0751ea20434d4ea7905fd9b1b9720a7f8488289c505e601216ffcd"}
Nov 01 20:06:23.551939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.551887    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podStartSLOduration=1.551868978 podStartE2EDuration="1.551868978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:23.549342333 +0000 UTC m=+428.964974807" watchObservedRunningTime="2026-11-01 20:06:23.551868978 +0000 UTC m=+428.967501451"
Nov 01 20:06:23.568850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:23.568752    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podStartSLOduration=1.568738909 podStartE2EDuration="1.568738909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:23.567263613 +0000 UTC m=+428.982896086" watchObservedRunningTime="2026-11-01 20:06:23.568738909 +0000 UTC m=+428.984371381"
Nov 01 20:06:24.029547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:24.029503    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:24.031027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:24.031001    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:24.031131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:24.031047    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:24.073671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:24.057128    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:24.073671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:24.057287    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:24.073671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:24.057342    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:25.029328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.029294    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:25.029499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.029355    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:25.055982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.055951    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:25.056519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.056488    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:25.392231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.392138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:25.475684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.471705    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:25.789672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.789100    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"]
Nov 01 20:06:25.806682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.805545    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.821862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.817039    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:25.821862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.820312    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\""
Nov 01 20:06:25.821862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821635    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.821862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821711    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.821862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821796    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.822236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821869    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.822236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821895    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.822236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821913    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l52v\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.822236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821933    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.822236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.821975    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.822236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.822001    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.824203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.823463    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"]
Nov 01 20:06:25.918817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.918781    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.922775    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.922826    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923122    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923164    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923195    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l52v\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923231    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923262    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923325    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923364    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.923682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923403    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.924330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.923727    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.924330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.924012    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.924330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.924035    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.924582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.924563    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.926454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.926431    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.926539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.926506    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.940711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.940630    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l52v\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.943780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.942205    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:25.959373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.952195    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"]
Nov 01 20:06:25.970615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.970018    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"]
Nov 01 20:06:25.970615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:25.970144    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.011834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.011798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:26.025007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.018721    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e27ae7-openshift-default-dockercfg-855jv\""
Nov 01 20:06:26.032533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.032497    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:26.032788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.032756    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:26.077743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.059971    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:26.077743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.060023    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:26.107669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.106008    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:26.125134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125096    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcg4g\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125137    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125167    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125191    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125346    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125408    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125556    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125583    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.125708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.125635    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.132627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.132602    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:26.212813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.212214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227317    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227441    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcg4g\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227471    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227499    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227521    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227551    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227573    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227622    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227656    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.227932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.227817    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.228622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.228598    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.228720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.228690    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.229096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.229072    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.229306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.229272    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.230405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.230371    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.230892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.230872    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.255668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.255234    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcg4g\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.255668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.255611    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.343287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.342834    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:26.590671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.571219    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"]
Nov 01 20:06:26.626762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.626732    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:26.626845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.626801    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:26.626845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.626830    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:26.776615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.770074    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"]
Nov 01 20:06:26.798817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.795876    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:26.798817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.795958    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:26.798817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:26.796000    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:27.030195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.029998    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:27.030195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.030056    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:27.064686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.056019    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:27.064686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.056080    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:27.601886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.600926    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" event={"ID":"34e99777-54a2-4ff5-875f-8d88ef07caeb","Type":"ContainerStarted","Data":"4050a378770af736146167332d30d55928f8a1af18022011c55cf7be6de447e3"}
Nov 01 20:06:27.601886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.601003    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" event={"ID":"34e99777-54a2-4ff5-875f-8d88ef07caeb","Type":"ContainerStarted","Data":"d8d7fb324fe78b5984c9c72216f69498fbfff0ba90ab9565579d209ea74d3768"}
Nov 01 20:06:27.604134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.604097    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" event={"ID":"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8","Type":"ContainerStarted","Data":"7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"}
Nov 01 20:06:27.604279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.604161    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" event={"ID":"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8","Type":"ContainerStarted","Data":"ba2f92f51815869682974f22ad8481d7d2b79c6f3a637a83f82a08f25cad087b"}
Nov 01 20:06:27.623925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.623865    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podStartSLOduration=2.6238459389999997 podStartE2EDuration="2.623845939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:27.621445719 +0000 UTC m=+433.037078193" watchObservedRunningTime="2026-11-01 20:06:27.623845939 +0000 UTC m=+433.039478414"
Nov 01 20:06:27.640696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:27.640614    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podStartSLOduration=2.640596705 podStartE2EDuration="2.640596705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:27.638994876 +0000 UTC m=+433.054627350" watchObservedRunningTime="2026-11-01 20:06:27.640596705 +0000 UTC m=+433.056229179"
Nov 01 20:06:28.029020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.028984    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:28.029173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.029042    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:28.056083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.056042    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:28.056362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.056331    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:28.133043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.133008    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:28.137826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.137799    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:28.343906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.343818    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:28.348984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.348957    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:28.611587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.611498    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:28.612022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.611720    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:28.612267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.612245    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:28.612517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:28.612495    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:29.029049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.029009    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:29.029227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.029073    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:29.056142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.056100    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:29.056435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.056399    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:29.222145 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:06:29.222106    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:06:29.428270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.428178    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:29.533937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.533878    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:29.636579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.636531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:29.891942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.891840    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:29.989438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:29.989383    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:30.028949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:30.028913    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Nov 01 20:06:30.029104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:30.028970    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Nov 01 20:06:30.052751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:30.052713    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 01 20:06:30.052948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:30.052780    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 01 20:06:30.092571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:30.092516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:30.859374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:30.859325    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:31.031328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.031289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:31.490699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.490661    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"]
Nov 01 20:06:31.496614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.496582    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.522301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.521175    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"]
Nov 01 20:06:31.550624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.550582    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsf8k\" (UniqueName: \"kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.550829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.550792    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.550901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.550872    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.550960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.550903    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.552399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.552366    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:31.566380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.566354    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3a6480-serving-cert\""
Nov 01 20:06:31.652371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.652326    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.652541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.652387    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.652541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.652465    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsf8k\" (UniqueName: \"kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.652705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.652562    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.672198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.671508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:31.672346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.672237    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsf8k\" (UniqueName: \"kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.672815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.672785    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.674237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.674209    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.675003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.674976    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.757619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.757527    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"]
Nov 01 20:06:31.762125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.762097    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:31.779951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.778497    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"]
Nov 01 20:06:31.782074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.781991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:31.829125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.828849    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:31.865674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.860054    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e27ae7-dockercfg-f6jgt\""
Nov 01 20:06:31.865674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.860152    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e27ae7\""
Nov 01 20:06:31.871059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.869268    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhwj\" (UniqueName: \"kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj\") pod \"payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:31.871059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.869319    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:31.972224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.970773    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhwj\" (UniqueName: \"kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj\") pod \"payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:31.972224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.970825    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:31.972224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.971798    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:31.982288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:31.981921    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhwj\" (UniqueName: \"kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj\") pod \"payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:32.030508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.030036    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:06:32.030508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.030116    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:06:32.054496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.054450    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:06:32.054687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.054513    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:06:32.090943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.090875    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:32.108140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.108107    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:32.109498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.109462    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:32.111701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.111317    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct"
Nov 01 20:06:32.116977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.116936    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:32.117442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.117407    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:32.118718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.118702    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq"
Nov 01 20:06:32.173427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.173402    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"]
Nov 01 20:06:32.276513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.276468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:32.365010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.364929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:32.382703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.382628    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"]
Nov 01 20:06:32.385238 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:06:32.385207    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0841ea_4959_469d_a6c7_dfff4d489cb2.slice/crio-f17fe315d8dc4948631f86bbc9e25168d18b68153fa9170f79406daa9e90c303 WatchSource:0}: Error finding container f17fe315d8dc4948631f86bbc9e25168d18b68153fa9170f79406daa9e90c303: Status 404 returned error can't find the container with id f17fe315d8dc4948631f86bbc9e25168d18b68153fa9170f79406daa9e90c303
Nov 01 20:06:32.486754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.486718    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:32.637864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.637767    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" event={"ID":"824bbcf5-e5e1-45a4-958a-bedb8740ad3e","Type":"ContainerStarted","Data":"b825dcaa69824267c7e49270c79dd355cbd8322843d81f4d8da0e12b5fde5907"}
Nov 01 20:06:32.639100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:32.639063    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" event={"ID":"1d0841ea-4959-469d-a6c7-dfff4d489cb2","Type":"ContainerStarted","Data":"f17fe315d8dc4948631f86bbc9e25168d18b68153fa9170f79406daa9e90c303"}
Nov 01 20:06:33.167867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:33.167791    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:33.263461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:33.263411    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:33.655672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:33.655278    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" event={"ID":"824bbcf5-e5e1-45a4-958a-bedb8740ad3e","Type":"ContainerStarted","Data":"2cc78417496074896773678a933f83b2d6b7b85330f282eff6d4f4f0833c31d2"}
Nov 01 20:06:33.673340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:33.673279    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" podStartSLOduration=2.111449996 podStartE2EDuration="2.67326425s" totalImagesPullingTime="561.814254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:33.671701589 +0000 UTC m=+439.087334063" watchObservedRunningTime="2026-11-01 20:06:33.67326425 +0000 UTC m=+439.088896722"
Nov 01 20:06:33.871534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:33.871497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:33.970997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:33.970946    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:34.071158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:34.071071    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:34.345573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:34.345498    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:06:34.626125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:34.626023    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:34.680423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:34.680391    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:35.263865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:35.263831    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:35.687715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:35.687617    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" event={"ID":"1d0841ea-4959-469d-a6c7-dfff4d489cb2","Type":"ContainerStarted","Data":"bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f"}
Nov 01 20:06:35.688134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:35.687862    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:35.704015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:35.703965    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" podStartSLOduration=2.191247298 podStartE2EDuration="4.703948233s" totalImagesPullingTime="2.512700935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:35.702604359 +0000 UTC m=+441.118236833" watchObservedRunningTime="2026-11-01 20:06:35.703948233 +0000 UTC m=+441.119580707"
Nov 01 20:06:36.261342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:36.261306    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:36.680772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:36.680736    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:37.553322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:37.553289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:38.372668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:38.372278    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:39.112163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:39.112135    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:39.762957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:39.762922    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:06:39.910862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:39.910822    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:40.716792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:40.716733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:40.722598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:40.722574    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:06:41.539622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:41.539574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:42.380591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:42.380555    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:43.275613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:43.275576    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:43.916888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:43.916809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:44.131131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.131089    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:44.232288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.232257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:44.651872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.651776    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:06:44.652040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.651862    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:06:44.777308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.777269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:44.880571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.880529    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:44.996725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:44.996629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:45.411545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.411458    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:45.511951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.511919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:45.611945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.611895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:45.890237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.890201    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"]
Nov 01 20:06:45.900317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.900150    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:45.902132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.902108    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"]
Nov 01 20:06:45.911126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.909089    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"default-dockercfg-98hrf\""
Nov 01 20:06:45.911126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.909113    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"simulator-w0-3a6480-kserve-self-signed-certs\""
Nov 01 20:06:45.929737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.929698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:45.933810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.933784    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"kube-root-ca.crt\""
Nov 01 20:06:45.933916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:45.933812    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"openshift-service-ca.crt\""
Nov 01 20:06:46.012087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.012046    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.012499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.012132    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.012499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.012180    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.012499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.012254    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.012499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.012308    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg76\" (UniqueName: \"kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.012499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.012333    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.112959    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.113036    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.113076    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.113105    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.113141    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg76\" (UniqueName: \"kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.113169    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.113663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.113624    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.114728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.114680    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.115222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.115200    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.118113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.117208    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.118113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.117919    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.129964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.129882    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg76\" (UniqueName: \"kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76\") pod \"simulator-w0-3a6480-kserve-596895b97-ggct6\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") " pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.130612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.130580    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:46.227868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.227829    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:46.520344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.520314    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"]
Nov 01 20:06:46.523075 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:06:46.523042    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2c852ae_b370_4114_bcb1_1c5fd54e8328.slice/crio-420e7eb226ad618db958534c4c90df9fe1e18820fa83c022ab1a3005b310955b WatchSource:0}: Error finding container 420e7eb226ad618db958534c4c90df9fe1e18820fa83c022ab1a3005b310955b: Status 404 returned error can't find the container with id 420e7eb226ad618db958534c4c90df9fe1e18820fa83c022ab1a3005b310955b
Nov 01 20:06:46.697565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.697532    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:06:46.789717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.789239    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" event={"ID":"e2c852ae-b370-4114-bcb1-1c5fd54e8328","Type":"ContainerStarted","Data":"0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab"}
Nov 01 20:06:46.789717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.789315    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" event={"ID":"e2c852ae-b370-4114-bcb1-1c5fd54e8328","Type":"ContainerStarted","Data":"420e7eb226ad618db958534c4c90df9fe1e18820fa83c022ab1a3005b310955b"}
Nov 01 20:06:46.806708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:46.790210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:47.228534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.228496    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:47.511297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.511212    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podStartSLOduration=2.511198777 podStartE2EDuration="2.511198777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:46.814849044 +0000 UTC m=+452.230481528" watchObservedRunningTime="2026-11-01 20:06:47.511198777 +0000 UTC m=+452.926831261"
Nov 01 20:06:47.512487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.512467    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:06:47.591516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.591475    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:47.794875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.794785    6479 generic.go:356] "Generic (PLEG): container finished" podID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerID="4050a378770af736146167332d30d55928f8a1af18022011c55cf7be6de447e3" exitCode=0
Nov 01 20:06:47.794875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.794858    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" event={"ID":"34e99777-54a2-4ff5-875f-8d88ef07caeb","Type":"ContainerDied","Data":"4050a378770af736146167332d30d55928f8a1af18022011c55cf7be6de447e3"}
Nov 01 20:06:47.795326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.795304    6479 scope.go:122] "RemoveContainer" containerID="4050a378770af736146167332d30d55928f8a1af18022011c55cf7be6de447e3"
Nov 01 20:06:47.796576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.796547    6479 generic.go:356] "Generic (PLEG): container finished" podID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerID="7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470" exitCode=0
Nov 01 20:06:47.796681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.796623    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" event={"ID":"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8","Type":"ContainerDied","Data":"7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"}
Nov 01 20:06:47.796681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.796641    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:47.796777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.796740    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:47.796815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.796788    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:47.797132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.797119    6479 scope.go:122] "RemoveContainer" containerID="7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"
Nov 01 20:06:47.798206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.798183    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:47.798267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.798242    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:47.798315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:47.798283    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:06:48.586677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:48.585511    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:48.832691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:48.824246    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" event={"ID":"34e99777-54a2-4ff5-875f-8d88ef07caeb","Type":"ContainerStarted","Data":"f597d0ff225d22937cc1b5fb36d5d89e555c0f2b3dd63a97b897c770c88566f5"}
Nov 01 20:06:48.836729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:48.832986    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" event={"ID":"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8","Type":"ContainerStarted","Data":"b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe"}
Nov 01 20:06:49.133148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.133057    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:49.133465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.133444    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:49.133597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.133574    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:49.345035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.345001    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:49.345353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.345334    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 20:06:49.345479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.345456    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 20:06:49.652221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.652181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:49.738672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.736418    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:49.978549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:49.978516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:50.134672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.133356    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:50.134672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.133416    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:50.347728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.347057    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 20:06:50.347728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.347121    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 20:06:50.804574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.798152    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:06:50.804574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.798243    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:06:50.828017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.822905    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"]
Nov 01 20:06:50.910672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.887743    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:50.930671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.927856    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"]
Nov 01 20:06:50.930671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:50.927986    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.028872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.016437    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"openshift-service-ca.crt\""
Nov 01 20:06:51.028872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.016609    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"kube-root-ca.crt\""
Nov 01 20:06:51.028872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.016723    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"simulator-w2-e27ae7-kserve-self-signed-certs\""
Nov 01 20:06:51.028872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.016818    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"default-dockercfg-cqft8\""
Nov 01 20:06:51.065716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.058133    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.065716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.058241    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.065716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.058372    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.065716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.058451    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.065716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.058489    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.065716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.058515    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpwl\" (UniqueName: \"kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.155561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.155075    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:51.155561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.155140    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.160256    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.160340    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.160378    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.160404    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpwl\" (UniqueName: \"kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.160456    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.160514    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.161460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.161033    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.197674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.175998    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.197674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.179818    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.198961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.198897    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.222822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.222228    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.238531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.238462    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpwl\" (UniqueName: \"kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl\") pod \"simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") " pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.346066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.345987    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 20:06:51.346327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.346297    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 20:06:51.368667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.367267    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:51.554505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:51.554398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:52.064950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.064751    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"]
Nov 01 20:06:52.137696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.135776    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:52.137696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.135836    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:52.344727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.343956    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 20:06:52.344727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.344145    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 20:06:52.447634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.447585    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:52.592984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:52.592948    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:53.133266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.133233    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:53.133725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.133284    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:53.144583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.144549    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" event={"ID":"8f79ac23-e682-4678-827f-3fdf4b5523c7","Type":"ContainerStarted","Data":"32e70d55a99dd1d70ceb6b18699553c7983a6128cacbbfc3fea00421b4b37566"}
Nov 01 20:06:53.144739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.144600    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" event={"ID":"8f79ac23-e682-4678-827f-3fdf4b5523c7","Type":"ContainerStarted","Data":"cfb2e26b998b5dab91305d096bda922c319039f250537e24590aaefade828b07"}
Nov 01 20:06:53.165150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.165090    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podStartSLOduration=3.165069045 podStartE2EDuration="3.165069045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:53.162943599 +0000 UTC m=+458.578576074" watchObservedRunningTime="2026-11-01 20:06:53.165069045 +0000 UTC m=+458.580701520"
Nov 01 20:06:53.285671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.283567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:53.345052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.345018    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 20:06:53.345327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.345294    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 20:06:53.369540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:53.369508    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:06:54.014671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:54.014488    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:54.135670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:54.135033    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:54.135670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:54.135094    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:54.344412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:54.343904    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 20:06:54.344412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:54.343967    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 20:06:54.860671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:54.854914    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:55.133402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:55.133256    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 01 20:06:55.133402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:55.133340    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 01 20:06:55.590221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:55.582412    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:56.048701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.047023    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:56.048701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.048006    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:56.050287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.050268    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:06:56.147979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.147943    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:56.149356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.149331    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:56.165889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.165861    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:06:56.256669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.253186    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:56.481678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.471538    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:56.932710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:56.932591    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:57.248702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:57.248664    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:06:57.754137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:57.754104    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:58.375853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:58.375813    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:58.956333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:58.956299    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:06:59.156503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:59.156465    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:06:59.170664 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:06:59.170616    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:06:59.648004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:06:59.647968    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:00.360536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:00.360504    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:00.955707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:00.955666    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:01.368162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:01.368085    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:07:01.380848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:01.380821    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:07:01.475816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:01.475781    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:07:01.665159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:01.665051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:02.266829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:02.266791    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:02.538523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:02.538443    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:03.063846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:03.063816    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:03.660330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:03.660289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:04.188140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:04.188105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:04.288226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:04.288188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:04.657925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:04.657890    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:07:04.980788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:04.980752    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:05.742497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:05.742465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:06.444934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:06.444895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:07.241311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:07.241278    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:07.957879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:07.957847    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:08.059163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:08.059121    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:08.754948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:08.754912    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:09.432034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:09.431994    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:10.250538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.250505    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:10.687998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.687913    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"]
Nov 01 20:07:10.707018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.706975    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.714542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.713962    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"kube-root-ca.crt\""
Nov 01 20:07:10.718436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.715001    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"premium-w4-0aa61b-kserve-self-signed-certs\""
Nov 01 20:07:10.718436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.716012    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"default-dockercfg-dnkln\""
Nov 01 20:07:10.726689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.722235    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"openshift-service-ca.crt\""
Nov 01 20:07:10.729551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.729518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:10.730736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.730713    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"]
Nov 01 20:07:10.758688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.758634    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.758959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.758891    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.759088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.758978    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.759088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.759066    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.759211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.759140    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldlf\" (UniqueName: \"kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.759211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.759191    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.860914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.860871    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.861277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.861242    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldlf\" (UniqueName: \"kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.861443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.861423    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.861756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.861723    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.862071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.862052    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.862249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.862231    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.862415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.862378    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.862872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.862847    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.863980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.863675    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.867688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.867629    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.871292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.869539    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:10.878379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:10.878353    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldlf\" (UniqueName: \"kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf\") pod \"premium-w4-0aa61b-kserve-6b479d4c99-5cs2r\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") " pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:11.034766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.034729    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:11.071471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.071395    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:11.314450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.314412    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"]
Nov 01 20:07:11.316568 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:07:11.316540    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07174d89_b04b_47e6_8fd8_e8cfa611c75a.slice/crio-a17cb88843c6a9038e26efdaef74b98a3f0b2b176b873d86740ca081b535fae2 WatchSource:0}: Error finding container a17cb88843c6a9038e26efdaef74b98a3f0b2b176b873d86740ca081b535fae2: Status 404 returned error can't find the container with id a17cb88843c6a9038e26efdaef74b98a3f0b2b176b873d86740ca081b535fae2
Nov 01 20:07:11.470837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.470796    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:11.525658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.525614    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" event={"ID":"07174d89-b04b-47e6-8fd8-e8cfa611c75a","Type":"ContainerStarted","Data":"094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959"}
Nov 01 20:07:11.525820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.525666    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" event={"ID":"07174d89-b04b-47e6-8fd8-e8cfa611c75a","Type":"ContainerStarted","Data":"a17cb88843c6a9038e26efdaef74b98a3f0b2b176b873d86740ca081b535fae2"}
Nov 01 20:07:11.543226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.543182    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podStartSLOduration=1.543169333 podStartE2EDuration="1.543169333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:11.541776071 +0000 UTC m=+476.957408568" watchObservedRunningTime="2026-11-01 20:07:11.543169333 +0000 UTC m=+476.958801806"
Nov 01 20:07:11.900943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:11.900906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:12.019701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:12.019629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:12.035103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:12.035055    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:12.112884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:12.112846    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:12.653456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:12.653422    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:13.255187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:13.255153    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:14.117280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:14.117244    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:14.756092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:14.756043    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:14.868189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:14.868136    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:14.969725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:14.969678    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:14.971023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:14.970993    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:07:15.604627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:15.604598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:16.103536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.103492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:16.189708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.189637    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"]
Nov 01 20:07:16.194362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.194342    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.205695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.205668    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e3c4a7\""
Nov 01 20:07:16.209690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.209668    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e3c4a7-dockercfg-h4zrt\""
Nov 01 20:07:16.210963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.210934    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"]
Nov 01 20:07:16.219521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.219495    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"]
Nov 01 20:07:16.222928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.222907    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.223374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.223352    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.223503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.223484    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kw6\" (UniqueName: \"kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6\") pod \"payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.240610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.240582    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"]
Nov 01 20:07:16.271902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.271733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:16.326477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.326432    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4p7\" (UniqueName: \"kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7\") pod \"payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.328451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.328420    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.332899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.332876    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.333175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.333155    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kw6\" (UniqueName: \"kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6\") pod \"payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.337601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.337571    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.345702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.343327    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kw6\" (UniqueName: \"kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6\") pod \"payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.434249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.434165    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4p7\" (UniqueName: \"kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7\") pod \"payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.434403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.434286    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.435101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.435076    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.442746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.442711    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4p7\" (UniqueName: \"kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7\") pod \"payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.507274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.507242    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:16.533075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.533049    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:16.537071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.537044    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:16.751055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.751021    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:16.786107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.786080    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"]
Nov 01 20:07:16.788074 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:07:16.788032    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbd8abe_4cc0_4368_b6f2_cc4ea3e8062d.slice/crio-12f7440d76bd24dda9eed3f35c3c653f55e26d4141066a143cbde5c7aaaa1841 WatchSource:0}: Error finding container 12f7440d76bd24dda9eed3f35c3c653f55e26d4141066a143cbde5c7aaaa1841: Status 404 returned error can't find the container with id 12f7440d76bd24dda9eed3f35c3c653f55e26d4141066a143cbde5c7aaaa1841
Nov 01 20:07:16.871023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:16.870990    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"]
Nov 01 20:07:16.874706 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:07:16.874678    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb072add5_cfbc_480e_a954_df9adeae1686.slice/crio-dd85de24ee67903f8e535859d2fecffe95b347929eb32d541806005e355b485a WatchSource:0}: Error finding container dd85de24ee67903f8e535859d2fecffe95b347929eb32d541806005e355b485a: Status 404 returned error can't find the container with id dd85de24ee67903f8e535859d2fecffe95b347929eb32d541806005e355b485a
Nov 01 20:07:17.189530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.189454    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:17.294184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.294014    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:17.389949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.389893    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:17.561664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.561612    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" event={"ID":"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d","Type":"ContainerStarted","Data":"7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889"}
Nov 01 20:07:17.561896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.561735    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" event={"ID":"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d","Type":"ContainerStarted","Data":"12f7440d76bd24dda9eed3f35c3c653f55e26d4141066a143cbde5c7aaaa1841"}
Nov 01 20:07:17.561896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.561765    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:17.563261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.563234    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" event={"ID":"b072add5-cfbc-480e-a954-df9adeae1686","Type":"ContainerStarted","Data":"88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6"}
Nov 01 20:07:17.563404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.563275    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" event={"ID":"b072add5-cfbc-480e-a954-df9adeae1686","Type":"ContainerStarted","Data":"dd85de24ee67903f8e535859d2fecffe95b347929eb32d541806005e355b485a"}
Nov 01 20:07:17.563404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.563374    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:17.582105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.582055    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" podStartSLOduration=1.582042032 podStartE2EDuration="1.582042032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:17.580032559 +0000 UTC m=+482.995665031" watchObservedRunningTime="2026-11-01 20:07:17.582042032 +0000 UTC m=+482.997674505"
Nov 01 20:07:17.956723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:17.956691    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:18.642639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:18.642598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:19.244936 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:19.244901    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:19.858747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:19.858712    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:20.550968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:20.550933    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:21.041530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:21.040680    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:21.060686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:21.059502    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:21.079352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:21.079301    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" podStartSLOduration=5.079286839 podStartE2EDuration="5.079286839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:17.594930405 +0000 UTC m=+483.010562883" watchObservedRunningTime="2026-11-01 20:07:21.079286839 +0000 UTC m=+486.494919304"
Nov 01 20:07:21.343847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:21.343761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:21.445029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:21.444984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:21.593891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:21.593819    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:07:22.289362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:22.289327    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:23.159005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:23.158961    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:23.947930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:23.947895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:24.575248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:24.575209    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:25.165609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:25.165563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:25.765623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:25.765587    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:26.361905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:26.361863    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:26.767043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:26.767002    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:07:27.166434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:27.166339    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:28.075911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:28.075872    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:28.570357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:28.570321    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:07:28.570524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:28.570394    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:07:29.138880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:29.138847    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:29.169269 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:07:29.169229    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:07:29.771042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:29.771002    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:29.909182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:29.909148    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:30.204658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:30.204626    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:30.610784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:30.610685    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:30.770297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:30.770263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:30.878047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:30.877929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:31.637146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:31.637113    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:32.450299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:32.450269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:33.358456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:33.358421    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:33.962410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:33.962375    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:34.066429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:34.066388    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:34.390679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:34.390586    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:35.072303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.072269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:35.174592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.174560    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:35.218884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.218846    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"]
Nov 01 20:07:35.224313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.224291    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.231315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.231282    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"]
Nov 01 20:07:35.232303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.231386    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4a127b\""
Nov 01 20:07:35.232303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.231630    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4a127b-dockercfg-bxqcl\""
Nov 01 20:07:35.250461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.250434    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"]
Nov 01 20:07:35.254333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.254312    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.267106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.267079    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"]
Nov 01 20:07:35.270420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.270391    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:35.376701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.376583    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sgk\" (UniqueName: \"kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk\") pod \"payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.377105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.377075    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8vr\" (UniqueName: \"kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr\") pod \"payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.377391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.377368    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.377742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.377715    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.478552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.478514    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sgk\" (UniqueName: \"kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk\") pod \"payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.478552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.478555    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8vr\" (UniqueName: \"kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr\") pod \"payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.479025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.478595    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.479025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.478631    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.479398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.479378    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.479439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.479418    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.486974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.486948    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8vr\" (UniqueName: \"kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr\") pod \"payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.487093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.487038    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sgk\" (UniqueName: \"kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk\") pod \"payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.543438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.543399    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:35.572699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.572670    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:35.822323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.822293    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"]
Nov 01 20:07:35.824446 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:07:35.824416    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057dad34_dbe0_4647_b35b_e350bb0f80d6.slice/crio-e10b6be19fec60d8fd6d2c4d136f3787e542c5372b37eaf8b1b8232a944125cf WatchSource:0}: Error finding container e10b6be19fec60d8fd6d2c4d136f3787e542c5372b37eaf8b1b8232a944125cf: Status 404 returned error can't find the container with id e10b6be19fec60d8fd6d2c4d136f3787e542c5372b37eaf8b1b8232a944125cf
Nov 01 20:07:35.854929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.854902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:35.860711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:35.860689    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"]
Nov 01 20:07:35.870640 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:07:35.870617    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421010e1_3dec_4e2e_818c_3b42ed9a9f9f.slice/crio-28f0dc65724e85169ee10a6fad31a395e07100b2540c81c00803bb40b6642759 WatchSource:0}: Error finding container 28f0dc65724e85169ee10a6fad31a395e07100b2540c81c00803bb40b6642759: Status 404 returned error can't find the container with id 28f0dc65724e85169ee10a6fad31a395e07100b2540c81c00803bb40b6642759
Nov 01 20:07:36.529320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.529239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:36.665021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.664985    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" event={"ID":"421010e1-3dec-4e2e-818c-3b42ed9a9f9f","Type":"ContainerStarted","Data":"02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c"}
Nov 01 20:07:36.665180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.665041    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" event={"ID":"421010e1-3dec-4e2e-818c-3b42ed9a9f9f","Type":"ContainerStarted","Data":"28f0dc65724e85169ee10a6fad31a395e07100b2540c81c00803bb40b6642759"}
Nov 01 20:07:36.665180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.665096    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:36.666724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.666691    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" event={"ID":"057dad34-dbe0-4647-b35b-e350bb0f80d6","Type":"ContainerStarted","Data":"5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae"}
Nov 01 20:07:36.666849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.666740    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" event={"ID":"057dad34-dbe0-4647-b35b-e350bb0f80d6","Type":"ContainerStarted","Data":"e10b6be19fec60d8fd6d2c4d136f3787e542c5372b37eaf8b1b8232a944125cf"}
Nov 01 20:07:36.666849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.666774    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:36.682311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.682257    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" podStartSLOduration=1.682241641 podStartE2EDuration="1.682241641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:36.68042221 +0000 UTC m=+502.096054694" watchObservedRunningTime="2026-11-01 20:07:36.682241641 +0000 UTC m=+502.097874117"
Nov 01 20:07:36.696594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:36.696511    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" podStartSLOduration=1.696499825 podStartE2EDuration="1.696499825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:36.695900938 +0000 UTC m=+502.111533412" watchObservedRunningTime="2026-11-01 20:07:36.696499825 +0000 UTC m=+502.112132298"
Nov 01 20:07:37.200226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.200190    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"]
Nov 01 20:07:37.203857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.203837    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.210493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.210465    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4a127b-serving-cert\""
Nov 01 20:07:37.213548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.213518    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"]
Nov 01 20:07:37.247130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.247076    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:37.297190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.297152    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.297353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.297218    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhdm\" (UniqueName: \"kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.297391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.297363    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.297422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.297393    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.353678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.351399    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:37.398226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.398194    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.398379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.398234    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.398379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.398295    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.398379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.398364    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhdm\" (UniqueName: \"kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.398979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.398955    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.400836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.400817    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.400921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.400851    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.405948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.405928    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhdm\" (UniqueName: \"kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm\") pod \"maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.445115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.445084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:37.519966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.519891    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:37.803885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.800411    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"]
Nov 01 20:07:37.808573 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:07:37.808542    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfbe007_29f2_47c0_bf77_db7325485d77.slice/crio-ac96c6cf49239d54447f87a95eebb9404c396f292626c4ca18377c8becaeb2a9 WatchSource:0}: Error finding container ac96c6cf49239d54447f87a95eebb9404c396f292626c4ca18377c8becaeb2a9: Status 404 returned error can't find the container with id ac96c6cf49239d54447f87a95eebb9404c396f292626c4ca18377c8becaeb2a9
Nov 01 20:07:37.964284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:37.964245    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:38.639941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:38.639857    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:38.678596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:38.678556    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt" event={"ID":"acfbe007-29f2-47c0-bf77-db7325485d77","Type":"ContainerStarted","Data":"5ca17e9dd3cefa84d4f784b6127aaa1ba47901efb8d5d2d3e19b32e9b350e84a"}
Nov 01 20:07:38.678774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:38.678623    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:38.678774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:38.678639    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt" event={"ID":"acfbe007-29f2-47c0-bf77-db7325485d77","Type":"ContainerStarted","Data":"ac96c6cf49239d54447f87a95eebb9404c396f292626c4ca18377c8becaeb2a9"}
Nov 01 20:07:38.719673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:38.715077    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt" podStartSLOduration=1.236860549 podStartE2EDuration="1.715054997s" totalImagesPullingTime="478.194448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:38.708101223 +0000 UTC m=+504.123733697" watchObservedRunningTime="2026-11-01 20:07:38.715054997 +0000 UTC m=+504.130687473"
Nov 01 20:07:39.328155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:39.328124    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:39.985914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:39.985879    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:40.648288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:40.648253    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:07:40.939844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:40.939758    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:41.789464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:41.789427    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:42.694378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:42.694338    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:44.062389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:44.062355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:44.689590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:44.689553    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:07:45.251510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:45.251477    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:46.232778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:46.232743    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:46.392330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:46.392297    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:47.643281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:47.643247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:47.674345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:47.674321    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:07:47.674500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:47.674386    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:07:48.573456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:48.573420    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:49.567336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:49.567296    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:50.379246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:50.379211    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:51.165823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:51.165782    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:51.720263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:51.720224    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:07:51.849305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:51.849272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:52.803112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:52.803065    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:53.795759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:53.795726    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:54.049668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:54.049501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:54.496145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:54.496099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:55.233984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:55.233955    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:55.451159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:55.451126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:56.168378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:56.168340    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:56.966378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:56.966343    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:57.731101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:57.731067    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:58.581391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:58.581347    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:07:59.170440 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:07:59.170397    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:07:59.568593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:07:59.568555    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:00.148020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.147987    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:00.662454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.662420    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"]
Nov 01 20:08:00.665941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.665922    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.679820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.679790    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4a127b\"/\"kube-root-ca.crt\""
Nov 01 20:08:00.679820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.679818    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4a127b\"/\"test-model-4a127b-kserve-self-signed-certs\""
Nov 01 20:08:00.680222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.679870    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4a127b\"/\"openshift-service-ca.crt\""
Nov 01 20:08:00.680222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.679914    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4a127b\"/\"default-dockercfg-hjdtn\""
Nov 01 20:08:00.680322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.680301    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"]
Nov 01 20:08:00.702265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.702228    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:00.780568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.780531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:00.782021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.781978    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.782179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.782152    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.782311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.782284    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.782420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.782340    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.782420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.782415    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.782539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.782465    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgms\" (UniqueName: \"kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.887686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.887319    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.887686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.887414    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.887686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.887483    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.887686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.887524    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgms\" (UniqueName: \"kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.887686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.887607    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.888103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.887717    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.891670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.888195    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.891670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.888471    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.891670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.888793    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.898004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.897900    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.900712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.900684    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgms\" (UniqueName: \"kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.904425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.904398    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs\") pod \"test-model-4a127b-kserve-7c4fb87f7c-z59gf\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") " pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:00.981442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:00.981405    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:01.261943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:01.261914    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"]
Nov 01 20:08:01.264044 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:01.264016    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9486b112_3b6b_4bbf_9cb8_b032c585eb63.slice/crio-eceb3ae6dc296d0361bba3720681370cd946e643a53f46e2f5343b8a0976d5f5 WatchSource:0}: Error finding container eceb3ae6dc296d0361bba3720681370cd946e643a53f46e2f5343b8a0976d5f5: Status 404 returned error can't find the container with id eceb3ae6dc296d0361bba3720681370cd946e643a53f46e2f5343b8a0976d5f5
Nov 01 20:08:01.649449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:01.649346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:01.807730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:01.807682    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" event={"ID":"9486b112-3b6b-4bbf-9cb8-b032c585eb63","Type":"ContainerStarted","Data":"265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e"}
Nov 01 20:08:01.807905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:01.807742    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" event={"ID":"9486b112-3b6b-4bbf-9cb8-b032c585eb63","Type":"ContainerStarted","Data":"eceb3ae6dc296d0361bba3720681370cd946e643a53f46e2f5343b8a0976d5f5"}
Nov 01 20:08:01.828693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:01.827971    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podStartSLOduration=1.827950506 podStartE2EDuration="1.827950506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:01.82493991 +0000 UTC m=+527.240572393" watchObservedRunningTime="2026-11-01 20:08:01.827950506 +0000 UTC m=+527.243582981"
Nov 01 20:08:01.981891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:01.981849    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:02.369457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:02.369350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:03.167477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.167436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:03.774980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.774947    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"]
Nov 01 20:08:03.780050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.780020    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.785839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.785814    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"distinct-2-w4-0aa61b-kserve-self-signed-certs\""
Nov 01 20:08:03.789053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.789001    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"]
Nov 01 20:08:03.792507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.791945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:03.818159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.818127    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.818331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.818176    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.818331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.818303    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rqd\" (UniqueName: \"kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.818401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.818361    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.818472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.818453    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.818504 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.818491    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.886101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.886064    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.922770    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.922877    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.922924    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.923030    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.923080    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.923188    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rqd\" (UniqueName: \"kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.924025    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.925684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.924270    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.926921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.926580    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.932737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.932675    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.949628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.949098    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:03.969424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:03.958363    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rqd\" (UniqueName: \"kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd\") pod \"distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:04.105034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:04.104929    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:04.389024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:04.388998    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"]
Nov 01 20:08:04.392236 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:04.392202    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13f5dfca_b212_43a2_b3ba_4e316cdbbd71.slice/crio-d9d151d7b90e93cf748a1f94e910fc88bb5b830d48074a7ae2560f897846c44b WatchSource:0}: Error finding container d9d151d7b90e93cf748a1f94e910fc88bb5b830d48074a7ae2560f897846c44b: Status 404 returned error can't find the container with id d9d151d7b90e93cf748a1f94e910fc88bb5b830d48074a7ae2560f897846c44b
Nov 01 20:08:04.552529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:04.552491    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:04.825879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:04.825839    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" event={"ID":"13f5dfca-b212-43a2-b3ba-4e316cdbbd71","Type":"ContainerStarted","Data":"322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908"}
Nov 01 20:08:04.826367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:04.825890    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" event={"ID":"13f5dfca-b212-43a2-b3ba-4e316cdbbd71","Type":"ContainerStarted","Data":"d9d151d7b90e93cf748a1f94e910fc88bb5b830d48074a7ae2560f897846c44b"}
Nov 01 20:08:04.842589 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:04.842536    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podStartSLOduration=1.842522104 podStartE2EDuration="1.842522104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:04.841354372 +0000 UTC m=+530.256986856" watchObservedRunningTime="2026-11-01 20:08:04.842522104 +0000 UTC m=+530.258154577"
Nov 01 20:08:05.106168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:05.106077    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:05.261140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:05.261103    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:05.489415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:05.489364    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:08:06.029164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:06.029126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:06.870380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:06.870342    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:07.765532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:07.765495    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:08.573289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:08.573247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:09.244601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:09.244568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:09.853281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:09.853243    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:10.452017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:10.451971    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:10.982006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:10.981972    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:11.000592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:11.000552    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:11.266310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:11.266222    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:11.887041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:11.886741    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:08:11.982297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:11.981901    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:12.653558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:12.653511    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:13.248234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:13.248193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:13.856157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:13.856103    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:14.105968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:14.105920    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:14.119080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:14.118997    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:14.478810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:14.478773    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:14.904322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:14.904234    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:08:15.252557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:15.252515    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:15.859545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:15.859493    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:15.863890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:15.863861    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:08:16.930843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:16.930809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:17.560732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:17.560698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:18.261725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:18.261693    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:18.934785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:18.934753    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:19.644453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:19.644421    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:20.266573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:20.266539    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:20.929355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:20.929314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:21.384247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:21.384210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:21.645571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:21.645487    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:22.355724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:22.355683    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:23.046041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:23.045992    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:23.660068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:23.659997    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:24.258257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:24.258217    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:24.935427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:24.935345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:25.547618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:25.547580    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:26.029991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.029953    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"]
Nov 01 20:08:26.035415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.035378    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.044189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.044156    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"]
Nov 01 20:08:26.045935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.045906    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3a6480-openshift-default-dockercfg-7wlx7\""
Nov 01 20:08:26.091858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.091816    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.091894    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.091997    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.092032    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.092049    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.092175    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmc8\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.092221    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.092237    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.092383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.092363    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.142106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.142061    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"]
Nov 01 20:08:26.146555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.146526    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.159126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.159089    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"]
Nov 01 20:08:26.189104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.189066    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:08:26.193178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193144    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.193514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193199    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.193514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193381    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqln7\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.193514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193444    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.193514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193476    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.193815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193536    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.193815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193568    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.193815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193617    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.193815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193678    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.193815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193708    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.193815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193773    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193817    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193852    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193903    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmc8\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193955    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.193992    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.194026    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.194035    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.194126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.194068    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.194535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.194363    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.194630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.194607    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.194935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.194909    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.195199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.195181    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.198389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.198353    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.203778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.203753    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.203959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.203938    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmc8\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.207274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.207240    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.223881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.223849    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300291    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300393    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300453    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqln7\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300497    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300548    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300604    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300641    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300694    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.300756    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.301224    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.301498    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.303842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.301753    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.312701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.311752    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.312893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.312822    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.316041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.316013    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.316912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.316886    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.318503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.318422    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqln7\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.325967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.325922    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.351965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.351931    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:26.463947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.463905    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:26.640291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.640265    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"]
Nov 01 20:08:26.641860 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:26.641835    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af719b3_4af3_4cd2_86cd_d23723c7660b.slice/crio-be5292098d6031ca1be5263b70fa64abbe9cb8ae3e1c04fcc6969db90bdead42 WatchSource:0}: Error finding container be5292098d6031ca1be5263b70fa64abbe9cb8ae3e1c04fcc6969db90bdead42: Status 404 returned error can't find the container with id be5292098d6031ca1be5263b70fa64abbe9cb8ae3e1c04fcc6969db90bdead42
Nov 01 20:08:26.644216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.644183    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:26.644292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.644252    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:26.644292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.644282    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:26.750523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.750488    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"]
Nov 01 20:08:26.754851 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:26.754819    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a7451a_fc38_4af7_bace_2d2c11463a48.slice/crio-7ebb8b507dc04a28fa6e8ec60bdc1715588b7b1be0c45cebb405b0dc9e65ac79 WatchSource:0}: Error finding container 7ebb8b507dc04a28fa6e8ec60bdc1715588b7b1be0c45cebb405b0dc9e65ac79: Status 404 returned error can't find the container with id 7ebb8b507dc04a28fa6e8ec60bdc1715588b7b1be0c45cebb405b0dc9e65ac79
Nov 01 20:08:26.758043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.758000    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:26.758145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.758089    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:26.758145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.758135    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:26.883762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.883590    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:26.996733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.995529    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" event={"ID":"8af719b3-4af3-4cd2-86cd-d23723c7660b","Type":"ContainerStarted","Data":"d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d"}
Nov 01 20:08:26.996733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:26.995610    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" event={"ID":"8af719b3-4af3-4cd2-86cd-d23723c7660b","Type":"ContainerStarted","Data":"be5292098d6031ca1be5263b70fa64abbe9cb8ae3e1c04fcc6969db90bdead42"}
Nov 01 20:08:27.044719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.044193    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" event={"ID":"f2a7451a-fc38-4af7-bace-2d2c11463a48","Type":"ContainerStarted","Data":"837c27fe81505faf2276e492d9c91897b7f068e3dd3e0cbb21d4879740b74f06"}
Nov 01 20:08:27.044719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.044265    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" event={"ID":"f2a7451a-fc38-4af7-bace-2d2c11463a48","Type":"ContainerStarted","Data":"7ebb8b507dc04a28fa6e8ec60bdc1715588b7b1be0c45cebb405b0dc9e65ac79"}
Nov 01 20:08:27.083745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.083679    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podStartSLOduration=1.08365958 podStartE2EDuration="1.08365958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:27.08220286 +0000 UTC m=+552.497835339" watchObservedRunningTime="2026-11-01 20:08:27.08365958 +0000 UTC m=+552.499292050"
Nov 01 20:08:27.086093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.086043    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podStartSLOduration=1.086030623 podStartE2EDuration="1.086030623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:27.022512667 +0000 UTC m=+552.438145140" watchObservedRunningTime="2026-11-01 20:08:27.086030623 +0000 UTC m=+552.501663096"
Nov 01 20:08:27.372818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.372717    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:27.379218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.378174    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:27.379218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.378238    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:27.466670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.466382    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:27.466670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.466518    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:27.466670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.466563    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:27.976669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:27.976153    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:28.364761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:28.363761    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:28.364761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:28.363824    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:28.468183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:28.468147    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:28.468477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:28.468445    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:28.786604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:28.786565    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:29.223249 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:08:29.223207    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:08:29.356669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:29.356060    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:29.356669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:29.356116    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:29.473727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:29.472790    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:29.473727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:29.472871    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:29.546606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:29.545820    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:30.288671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:30.286517    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:30.355674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:30.353033    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:30.355674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:30.353098    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:30.473672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:30.467042    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:30.473672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:30.467108    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:31.074671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:31.064035    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:31.354842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:31.354749    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:31.355131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:31.355091    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:31.467680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:31.466797    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:31.467680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:31.466857    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:31.792687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:31.792640    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:32.356105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:32.356064    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:32.356373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:32.356337    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:32.468672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:32.465033    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:32.468672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:32.465105    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:32.587614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:32.587564    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:33.254289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:33.254237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:33.353907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:33.353867    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 20:08:33.354897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:33.354862    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 20:08:33.465296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:33.465230    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 20:08:33.466153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:33.465595    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 20:08:33.953430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:33.953392    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:34.357891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.357802    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:34.358332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.358314    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:34.359193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.359167    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:08:34.469215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.469183    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:34.469606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.469581    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:34.470494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.470468    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:08:34.623868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:34.623785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:35.466963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:35.466926    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:35.570638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:35.570590    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:36.344899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:36.344866    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:36.955088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:36.955056    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:37.663012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:37.662978    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:38.449911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:38.449872    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:39.151212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:39.151180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:39.742709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:39.742676    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:40.455361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:40.455325    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:41.246842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:41.246809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:41.962612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:41.962572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:42.709690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:42.709627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:43.570374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:43.570328    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:44.073914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:44.073880    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:08:44.284431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:44.284393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:45.250144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:45.250103    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:46.488295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:46.488252    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:46.891624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:46.891525    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:47.303448 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:47.303403    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:47.400476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:47.400441    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:47.584271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:47.584182    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:48.128338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:48.128305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:48.861993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:48.861958    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:49.560912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:49.560880    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:50.558434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:50.558396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:51.232675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:51.232612    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:51.338806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:51.338766    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:51.436177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:51.436138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:52.167520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.167483    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:52.497014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.496974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:52.546814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.546781    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"]
Nov 01 20:08:52.552154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.552126    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.558165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.558124    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"]
Nov 01 20:08:52.560976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.560868    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e38ef4\""
Nov 01 20:08:52.560976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.560946    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e38ef4-dockercfg-6znv4\""
Nov 01 20:08:52.562019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.561995    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.563926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.563430    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"]
Nov 01 20:08:52.572057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.572025    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"]
Nov 01 20:08:52.576564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.576533    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmlwm\" (UniqueName: \"kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm\") pod \"payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.576957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.576934    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.596708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.596627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:52.688672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.678754    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqth\" (UniqueName: \"kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth\") pod \"payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.689028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.688994    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.689269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.689252    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmlwm\" (UniqueName: \"kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm\") pod \"payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.689484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.689468    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.711670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.690770    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.722438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.722386    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmlwm\" (UniqueName: \"kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm\") pod \"payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.790474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.790366    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.790688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.790471    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqth\" (UniqueName: \"kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth\") pod \"payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.791342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.791318    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.797956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.797921    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqth\" (UniqueName: \"kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth\") pod \"payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:52.880089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.880058    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:52.895778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:52.895752    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:53.058811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.058694    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:53.173324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.173296    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"]
Nov 01 20:08:53.176458 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:53.176430    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1359fa9e_f0ad_4b34_8c98_4361fef33f7f.slice/crio-687c48cf9c55612434c699d30cfee2a3591a59c38571f0a6417dfe11da89cbb6 WatchSource:0}: Error finding container 687c48cf9c55612434c699d30cfee2a3591a59c38571f0a6417dfe11da89cbb6: Status 404 returned error can't find the container with id 687c48cf9c55612434c699d30cfee2a3591a59c38571f0a6417dfe11da89cbb6
Nov 01 20:08:53.193620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.193593    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"]
Nov 01 20:08:53.196279 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:53.196247    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9cd2de_8613_46c3_8239_8853debf1530.slice/crio-05c2b5780fd9c86869105adb745120a91f1bdedcf1cf6d72edc1652f2ec53496 WatchSource:0}: Error finding container 05c2b5780fd9c86869105adb745120a91f1bdedcf1cf6d72edc1652f2ec53496: Status 404 returned error can't find the container with id 05c2b5780fd9c86869105adb745120a91f1bdedcf1cf6d72edc1652f2ec53496
Nov 01 20:08:53.340207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.340118    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" event={"ID":"dc9cd2de-8613-46c3-8239-8853debf1530","Type":"ContainerStarted","Data":"890fb8114b87f9dc6805c7269712868e7ec7be40feeee8bf0c7bf57a9d51e5c9"}
Nov 01 20:08:53.340207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.340174    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" event={"ID":"dc9cd2de-8613-46c3-8239-8853debf1530","Type":"ContainerStarted","Data":"05c2b5780fd9c86869105adb745120a91f1bdedcf1cf6d72edc1652f2ec53496"}
Nov 01 20:08:53.340432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.340241    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:08:53.342005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.341981    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" event={"ID":"1359fa9e-f0ad-4b34-8c98-4361fef33f7f","Type":"ContainerStarted","Data":"278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3"}
Nov 01 20:08:53.342139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.342021    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" event={"ID":"1359fa9e-f0ad-4b34-8c98-4361fef33f7f","Type":"ContainerStarted","Data":"687c48cf9c55612434c699d30cfee2a3591a59c38571f0a6417dfe11da89cbb6"}
Nov 01 20:08:53.342139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.342051    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:08:53.358843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.358791    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" podStartSLOduration=1.358773704 podStartE2EDuration="1.358773704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:53.354892764 +0000 UTC m=+578.770525238" watchObservedRunningTime="2026-11-01 20:08:53.358773704 +0000 UTC m=+578.774406178"
Nov 01 20:08:53.420558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.420506    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" podStartSLOduration=1.4204922500000001 podStartE2EDuration="1.42049225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:53.418376912 +0000 UTC m=+578.834009385" watchObservedRunningTime="2026-11-01 20:08:53.42049225 +0000 UTC m=+578.836124722"
Nov 01 20:08:53.847794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:53.847753    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:54.454837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:54.454803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:55.381665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:55.380010    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:55.514672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:55.499679    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:55.582953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:55.582423    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:55.961382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:55.961339    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"]
Nov 01 20:08:55.971149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:55.971050    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:55.983829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:55.983734    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"]
Nov 01 20:08:56.046421 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046387    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046436    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwvw\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046554    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046610    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046722    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046777    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046874    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.046969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.046923    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.047044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.047029    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.147835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.147796    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.147847    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.147875    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148027    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148079    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148123    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwvw\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148196    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148260    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148294    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148303    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148398    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148607    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.148639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.148611    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.149271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.149245    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.150300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.150268    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.150964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.150941    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.156144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.156117    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwvw\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.156244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.156229    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.310434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.310346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:56.317343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.317316    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:56.668140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.668106    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"]
Nov 01 20:08:56.669973 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:08:56.669945    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093e6b04_5681_4f90_b80e_f7b0fbc7ce17.slice/crio-c0b57ba8beb0ff443b8dffaed3ed9258bdfa1a60f87202be6c19854927783fce WatchSource:0}: Error finding container c0b57ba8beb0ff443b8dffaed3ed9258bdfa1a60f87202be6c19854927783fce: Status 404 returned error can't find the container with id c0b57ba8beb0ff443b8dffaed3ed9258bdfa1a60f87202be6c19854927783fce
Nov 01 20:08:56.672861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.672816    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:56.672943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.672914    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:56.672981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:56.672963    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:08:57.056992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:57.056956    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:57.439884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:57.439794    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" event={"ID":"093e6b04-5681-4f90-b80e-f7b0fbc7ce17","Type":"ContainerStarted","Data":"302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74"}
Nov 01 20:08:57.439884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:57.439848    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" event={"ID":"093e6b04-5681-4f90-b80e-f7b0fbc7ce17","Type":"ContainerStarted","Data":"c0b57ba8beb0ff443b8dffaed3ed9258bdfa1a60f87202be6c19854927783fce"}
Nov 01 20:08:57.744014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:57.743980    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:58.317839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:58.317800    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:08:58.319595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:58.319567    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:08:58.319745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:58.319629    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 20:08:58.545577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:58.545537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:59.147853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:59.147822    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:59.198259 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:08:59.198213    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:08:59.318418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:59.318373    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:08:59.318587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:59.318437    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 20:08:59.750251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:59.750200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:08:59.864669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:08:59.862536    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:00.317925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:00.317893    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:09:00.318356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:00.317946    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 20:09:00.377236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:00.377191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:01.051555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:01.051514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:01.318434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:01.318363    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:09:01.318937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:01.318807    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 20:09:01.658725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:01.657260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:02.318380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:02.318341    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:09:02.318557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:02.318395    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 20:09:02.351075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:02.351041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:03.044433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:03.044397    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:03.318042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:03.317955    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 01 20:09:03.318042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:03.318021    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 01 20:09:03.675603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:03.675514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:04.259615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.259575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:04.322754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.322723    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:09:04.323119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.323102    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:09:04.323826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.323810    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:09:04.346117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.346066    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podStartSLOduration=9.346049935 podStartE2EDuration="9.346049935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:57.483042077 +0000 UTC m=+582.898674550" watchObservedRunningTime="2026-11-01 20:09:04.346049935 +0000 UTC m=+589.761682412"
Nov 01 20:09:04.349972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.349947    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:09:04.350107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.350020    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:09:04.358960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:04.358936    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:09:05.518451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:05.518408    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:06.840856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:06.840817    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:07.623233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:07.623200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:08.486231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:08.486188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:09.186755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:09.186722    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:09.974404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:09.974367    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:10.776098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:10.776053    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:11.099628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.099541    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"]
Nov 01 20:09:11.105475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.105452    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.116283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.116248    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"]
Nov 01 20:09:11.128984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.128063    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"]
Nov 01 20:09:11.133656 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.133621    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.153032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.152969    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"]
Nov 01 20:09:11.187996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.187955    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6t6n\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.187996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188006    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188151    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188177    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188205    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188341    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188420    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188473    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188618    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188662    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188699    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkz76\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188732    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188786    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188833    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188861    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.189534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188892    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.189534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.188962    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.189534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.189013    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.227314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.227283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293529    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6t6n\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293584    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293695    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293742    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293773    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293800    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293839    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293866    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293920    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293942    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293973    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkz76\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.293999    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294050    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294095    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294123    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294160    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294194    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294220    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.294723    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.302676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.295414    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.311298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.307048    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.311298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.307995    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.311298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.308363    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.311298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.308663    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.311671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.311508    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.312227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.311825    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.312227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.312165    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.312417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.312282    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.312417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.312406    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.314118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.314090    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.315944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.315919    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.316397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.316376    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.316565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.316543    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6t6n\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.319528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.319497    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkz76\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.319890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.319864    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.321156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.321131    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.422463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.422371    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:11.451524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.451485    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:11.562380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.562304    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:11.706300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.706273    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"]
Nov 01 20:09:11.708783 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:09:11.708754    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f206db_9d47_4687_9cdc_884371b94797.slice/crio-11c3a40edac8ea93fe618243b1f78a6566feba12cc3b779f3a3cf6735e1ff5e3 WatchSource:0}: Error finding container 11c3a40edac8ea93fe618243b1f78a6566feba12cc3b779f3a3cf6735e1ff5e3: Status 404 returned error can't find the container with id 11c3a40edac8ea93fe618243b1f78a6566feba12cc3b779f3a3cf6735e1ff5e3
Nov 01 20:09:11.711725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.711695    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:11.711783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.711759    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:11.711826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.711788    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:11.758121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.758092    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"]
Nov 01 20:09:11.761256 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:09:11.761227    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4615003_28af_41a2_a79e_416ce58f2be7.slice/crio-1b5aa4461d778dc2e3b9231e4edfc022598c69f565c25402992d4f0491b30450 WatchSource:0}: Error finding container 1b5aa4461d778dc2e3b9231e4edfc022598c69f565c25402992d4f0491b30450: Status 404 returned error can't find the container with id 1b5aa4461d778dc2e3b9231e4edfc022598c69f565c25402992d4f0491b30450
Nov 01 20:09:11.764734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.764695    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:11.764811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.764791    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:11.764857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:11.764839    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:12.410954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.410912    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:12.508528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.508480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:12.664731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.656716    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" event={"ID":"f4615003-28af-41a2-a79e-416ce58f2be7","Type":"ContainerStarted","Data":"a132552396c0291abae65c540214b828b405786aa338494db644f46cfebd18ba"}
Nov 01 20:09:12.664731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.656789    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" event={"ID":"f4615003-28af-41a2-a79e-416ce58f2be7","Type":"ContainerStarted","Data":"1b5aa4461d778dc2e3b9231e4edfc022598c69f565c25402992d4f0491b30450"}
Nov 01 20:09:12.697675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.682934    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" event={"ID":"d0f206db-9d47-4687-9cdc-884371b94797","Type":"ContainerStarted","Data":"c4e12420e8c2f3e01f3afaf6f53ba919c11d7a526f153da11ac522b1138df298"}
Nov 01 20:09:12.697675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.683003    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" event={"ID":"d0f206db-9d47-4687-9cdc-884371b94797","Type":"ContainerStarted","Data":"11c3a40edac8ea93fe618243b1f78a6566feba12cc3b779f3a3cf6735e1ff5e3"}
Nov 01 20:09:12.811877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.806595    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podStartSLOduration=1.806574205 podStartE2EDuration="1.806574205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:12.804848654 +0000 UTC m=+598.220481128" watchObservedRunningTime="2026-11-01 20:09:12.806574205 +0000 UTC m=+598.222206680"
Nov 01 20:09:12.811877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:12.806799    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podStartSLOduration=1.806788332 podStartE2EDuration="1.806788332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:12.764931555 +0000 UTC m=+598.180564028" watchObservedRunningTime="2026-11-01 20:09:12.806788332 +0000 UTC m=+598.222420806"
Nov 01 20:09:13.091674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.082053    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:13.217828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.217787    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"]
Nov 01 20:09:13.557834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.557791    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:13.559260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.559224    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:13.619951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.619882    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:13.658107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.654619    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:13.658107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.657088    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"]
Nov 01 20:09:13.658107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.657141    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:13.658107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.657168    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:13.658107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.657261    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.682781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.681357    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5370bb-openshift-default-dockercfg-m26p7\""
Nov 01 20:09:13.856833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.856715    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.857171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.857137    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.857330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.857310    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.857498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.857479    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.857768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.857749    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.857910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.857895    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.858029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.858015    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh75t\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.858215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.858201    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.858334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.858320    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.873819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.873745    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:13.900864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.900805    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:09:13.961455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.961408    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.961805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.961764    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.962128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.961988    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.962398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.962370    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.962786    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.962845    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh75t\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.962922    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.962980    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.963009    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.963195    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.964193    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.982671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.974256    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:13.985385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.983385    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.003671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:13.997841    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.035672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.011052    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.035672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.033343    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.059670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.059561    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh75t\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.090675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.090339    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.154735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.150910    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:14.226452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.226375    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:14.298221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.298143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:14.348672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.344806    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:14.434293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.434172    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:14.434584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.434532    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:14.459682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.459614    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:14.459886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.459698    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:14.470607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.469744    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:14.595858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.595800    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:14.943182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.912863    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:14.943182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:14.922243    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 01 20:09:15.133899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.090522    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:09:15.133899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.090624    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:09:15.133899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.125771    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:15.350671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.350273    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:15.350879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.350854    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:09:15.427672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.424427    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:15.431667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.427927    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:15.470153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.470098    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:15.471028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.471002    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:09:15.533754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.532977    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:15.533754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.533040    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:15.556169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.554854    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:15.640791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.601148    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"]
Nov 01 20:09:15.640791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.601277    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:09:15.640791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.614220    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:09:15.683202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.671809    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:09:15.786714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.786665    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:15.788070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.788032    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:15.788265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.788244    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:15.794977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.794952    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:09:15.861765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.861733    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:09:15.942504 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.942421    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:09:15.943661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:15.943607    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:09:16.235608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:16.232979    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" event={"ID":"28deb188-880a-499b-8934-ad67970c7856","Type":"ContainerStarted","Data":"534bd32b9627460d406e9c7cdd3259c171a396afb8033270106c3b6e660a914a"}
Nov 01 20:09:16.326675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:16.326574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:16.428538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:16.428497    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:16.428861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:16.428832    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:16.473199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:16.473105    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:16.473468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:16.473436    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:17.207872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:17.207834    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:17.238514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:17.238474    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" event={"ID":"28deb188-880a-499b-8934-ad67970c7856","Type":"ContainerStarted","Data":"b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"}
Nov 01 20:09:17.442673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:17.431048    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:17.442673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:17.431114    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:17.455996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:17.455946    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:17.456288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:17.456248    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:18.152021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.151984    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:18.180999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.180968    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:18.213292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.213238    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:18.216149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.216090    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podStartSLOduration=6.216069526 podStartE2EDuration="6.216069526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:17.281016885 +0000 UTC m=+602.696649358" watchObservedRunningTime="2026-11-01 20:09:18.216069526 +0000 UTC m=+603.631702003"
Nov 01 20:09:18.244171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.244146    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:18.245195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.245173    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:18.429148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.429053    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:18.429412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.429380    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:18.500671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.500071    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:18.500671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:18.500148    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:19.396496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:19.396458    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:19.425458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:19.425421    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 20:09:19.425801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:19.425758    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 20:09:19.458671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:19.456250    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 20:09:19.458671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:19.456328    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 20:09:19.499242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:19.499207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:20.388701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:20.388627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:20.711934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:20.711894    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:20.834962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:20.834922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:21.425033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.424979    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:09:21.425235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.425051    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:21.445070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.445034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:21.502893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.502849    6479 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:09:21.503095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.502921    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:21.544944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.544904    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:21.548458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.548433    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:21.570670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.550016    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:21.571077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.571051    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:09:21.685489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.685404    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:21.686477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.686409    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:21.687748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:21.687478    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:09:22.274706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:22.274670    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:23.077040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:23.076998    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:23.883164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:23.883123    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:24.666483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:24.666446    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:24.779245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:24.779202    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:09:25.622670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:25.622337    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:25.763672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:25.763577    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:26.056674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.056000    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"]
Nov 01 20:09:26.106670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.095301    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.128683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.124770    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"]
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194494    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194577    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194615    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194713    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194835    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194857    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194889    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194948    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.202677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.194981    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7lq\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297327    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297373    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297413    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297476    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297502    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7lq\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297561    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297633    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297692    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.313737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.297763    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.322210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.322175    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.411760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.322687    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.411760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.323403    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.411760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.324030    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.411760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.324685    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.411760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.400685    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7lq\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.430872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.414793    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"]
Nov 01 20:09:26.529696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.515616    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.771706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.751464    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.811634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.774153    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.850410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.824044    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:26.850410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.831086    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:26.869890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.869865    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.885897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.885865    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"]
Nov 01 20:09:26.977725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977507    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.977725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977560    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.977725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977581    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.977725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977617    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.978074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977750    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.978074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977775    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.978074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977791    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.978074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977819    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jst\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:26.978074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:26.977864    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.064796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.064698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:27.078769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.078736    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.078955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.078846    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.078955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.078883    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.078955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.078908    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.078955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.078950    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jst\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079011    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079125    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079158    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079169    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079234    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079259    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079478    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079515    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.079921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.079896    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.081401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.081381    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.082090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.082063    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.086403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.086380    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.086537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.086515    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jst\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.180681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.179052    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"]
Nov 01 20:09:27.183342 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:09:27.183309    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41b185c_14cd_4e06_9e7c_63fd65f437ca.slice/crio-890deec4b6cd3891a793cfcea3f144c33b624de2d80d5de93dfb8770dcdd0963 WatchSource:0}: Error finding container 890deec4b6cd3891a793cfcea3f144c33b624de2d80d5de93dfb8770dcdd0963: Status 404 returned error can't find the container with id 890deec4b6cd3891a793cfcea3f144c33b624de2d80d5de93dfb8770dcdd0963
Nov 01 20:09:27.186435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.186355    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:27.186561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.186439    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:27.186561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.186480    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:27.210265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.210242    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:27.506709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.506678    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"]
Nov 01 20:09:27.511241 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:09:27.511200    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87573e4d_d137_4a92_aedb_c8e81bdfb470.slice/crio-1b1fcba4a792b951cff55ad61f73b2cca42daf69379ace2e7699c715813c25f6 WatchSource:0}: Error finding container 1b1fcba4a792b951cff55ad61f73b2cca42daf69379ace2e7699c715813c25f6: Status 404 returned error can't find the container with id 1b1fcba4a792b951cff55ad61f73b2cca42daf69379ace2e7699c715813c25f6
Nov 01 20:09:27.513836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.513803    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:27.513948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.513887    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:27.513948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.513919    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:27.812154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.812069    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" event={"ID":"87573e4d-d137-4a92-aedb-c8e81bdfb470","Type":"ContainerStarted","Data":"3e93cab6a575d9f260e7e526a0bf2906a437cb74e526fb5910e25a025c233d15"}
Nov 01 20:09:27.812154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.812128    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" event={"ID":"87573e4d-d137-4a92-aedb-c8e81bdfb470","Type":"ContainerStarted","Data":"1b1fcba4a792b951cff55ad61f73b2cca42daf69379ace2e7699c715813c25f6"}
Nov 01 20:09:27.814407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.814376    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" event={"ID":"d41b185c-14cd-4e06-9e7c-63fd65f437ca","Type":"ContainerStarted","Data":"956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831"}
Nov 01 20:09:27.814543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.814430    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" event={"ID":"d41b185c-14cd-4e06-9e7c-63fd65f437ca","Type":"ContainerStarted","Data":"890deec4b6cd3891a793cfcea3f144c33b624de2d80d5de93dfb8770dcdd0963"}
Nov 01 20:09:27.832726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.832664    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podStartSLOduration=1.832635072 podStartE2EDuration="1.832635072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:27.829165522 +0000 UTC m=+613.244797995" watchObservedRunningTime="2026-11-01 20:09:27.832635072 +0000 UTC m=+613.248267545"
Nov 01 20:09:27.833003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.832980    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:27.979555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:27.979153    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:28.210973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:28.210920    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:28.212123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:28.212084    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:28.212233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:28.212154    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:28.833505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:28.833471    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 20:09:28.833987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:28.833549    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 20:09:29.212950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:29.212787    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:29.212950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:29.212848    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:29.220876 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:09:29.220839    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:09:29.788671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:29.784512    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:29.836895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:29.836861    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 20:09:29.837503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:29.837469    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 20:09:30.210785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:30.210747    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:30.210964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:30.210853    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:30.832317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:30.832275    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 20:09:30.832494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:30.832350    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 20:09:31.105729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:31.095734    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:31.215207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:31.215159    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:31.215518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:31.215465    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:31.833290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:31.833252    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 20:09:31.833576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:31.833537    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 20:09:32.211303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:32.211264    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:32.211825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:32.211344    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:32.767673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:32.762100    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:32.837022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:32.836981    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 20:09:32.837199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:32.837042    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 20:09:33.213124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:33.213087    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:33.213698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:33.213669    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:33.833131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:33.833084    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 20:09:33.833312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:33.833157    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 20:09:34.211531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.211493    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 01 20:09:34.211726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.211597    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 01 20:09:34.636419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.636334    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:34.955518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.955487    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:34.956059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.955992    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:34.957248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.957226    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:09:34.981340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:34.981278    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podStartSLOduration=8.981257892 podStartE2EDuration="8.981257892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:27.856453087 +0000 UTC m=+613.272085560" watchObservedRunningTime="2026-11-01 20:09:34.981257892 +0000 UTC m=+620.396890365"
Nov 01 20:09:35.215222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:35.215138    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:35.215640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:35.215624    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:35.216292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:35.216275    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:09:35.915565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:35.915527    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:36.862378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:36.862346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:37.602125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:37.602090    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:39.012684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:39.012631    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:39.485617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:39.483410    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:39.938840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:39.938739    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:40.489310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.489208    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:40.610530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.610484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:40.724369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.724327    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:40.737761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.725748    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:40.820605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.816034    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:09:40.941675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.940765    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:40.941675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.941572    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:09:40.954667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:40.944265    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:41.017734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.004606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:41.188731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.187898    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:41.305820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.305780    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:41.390186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.390147    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:41.605212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.605122    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:41.736817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.736771    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:41.737840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.737816    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:09:41.890830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.871489    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:41.923724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.893374    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:09:41.923724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.910237    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:09:41.931424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.931309    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:09:41.939632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:41.939606    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:09:42.013367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.013333    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:42.117715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.116310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:42.687593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.687535    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:42.734220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.734185    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"]
Nov 01 20:09:42.742157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.739995    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:42.746817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.746790    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-142f6c4e991f\""
Nov 01 20:09:42.746971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.746863    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"openshift-service-ca.crt\""
Nov 01 20:09:42.746971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.746872    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"payload-processing-external-model-e2e--67d773b9-dockercfg-w8mwq\""
Nov 01 20:09:42.747067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.747005    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"kube-root-ca.crt\""
Nov 01 20:09:42.747199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.747165    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5370bb\""
Nov 01 20:09:42.748799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.748776    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"routing-overlay\""
Nov 01 20:09:42.749138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.749116    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"]
Nov 01 20:09:42.786953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.786925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:42.928712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.920543    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:42.928712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.920622    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:42.928712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.920710    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tkvn\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:42.936687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:42.936571    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.040222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.039419    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.041368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.041317    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.042019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.042000    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.043225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.043205    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tkvn\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.043788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.043769    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.044633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.044615    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.090086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.082821    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tkvn\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.090086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.083243    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.091600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.090949    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:43.431716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.431679    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"]
Nov 01 20:09:43.432919 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:09:43.432892    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7ffa7c_6b2a_4dec_84c0_90891a1fa5b2.slice/crio-6f08b4c2c8eb4a4c6b77e7b6990fceeaaa19ff6081eab8adf1f1d12c5919ad2f WatchSource:0}: Error finding container 6f08b4c2c8eb4a4c6b77e7b6990fceeaaa19ff6081eab8adf1f1d12c5919ad2f: Status 404 returned error can't find the container with id 6f08b4c2c8eb4a4c6b77e7b6990fceeaaa19ff6081eab8adf1f1d12c5919ad2f
Nov 01 20:09:43.528797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:43.528767    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:44.136100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:44.136062    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:44.308280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:44.308248    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n" event={"ID":"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2","Type":"ContainerStarted","Data":"0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316"}
Nov 01 20:09:44.308469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:44.308299    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n" event={"ID":"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2","Type":"ContainerStarted","Data":"6f08b4c2c8eb4a4c6b77e7b6990fceeaaa19ff6081eab8adf1f1d12c5919ad2f"}
Nov 01 20:09:44.309100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:44.309079    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:44.324837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:44.324784    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n" podStartSLOduration=2.324772597 podStartE2EDuration="2.324772597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:44.324087328 +0000 UTC m=+629.739719803" watchObservedRunningTime="2026-11-01 20:09:44.324772597 +0000 UTC m=+629.740405070"
Nov 01 20:09:44.849200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:44.849172    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:45.466671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:45.461207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:46.215403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:46.215362    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:46.930115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:46.930081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:47.677360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:47.677322    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:47.988289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:47.988247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:48.456404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:48.456369    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:49.069530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:49.069492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:49.286859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:49.286806    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:09:49.287041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:49.286880    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:49.856533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:49.856479    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:50.369523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.369486    6479 generic.go:356] "Generic (PLEG): container finished" podID="28deb188-880a-499b-8934-ad67970c7856" containerID="b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21" exitCode=0
Nov 01 20:09:50.369967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.369560    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" event={"ID":"28deb188-880a-499b-8934-ad67970c7856","Type":"ContainerDied","Data":"b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"}
Nov 01 20:09:50.370106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.370090    6479 scope.go:122] "RemoveContainer" containerID="b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"
Nov 01 20:09:50.371819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.371792    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:50.371916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.371886    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:50.371972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.371921    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:09:50.378270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.378249    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:09:50.753452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:50.753402    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:51.434675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:51.428817    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" event={"ID":"28deb188-880a-499b-8934-ad67970c7856","Type":"ContainerStarted","Data":"47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3"}
Nov 01 20:09:51.770669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:51.769613    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:52.156729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:52.155714    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:52.156729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:52.155880    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:09:52.156729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:52.155924    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 20:09:52.662119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:52.662082    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:53.155672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:53.152697    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:09:53.155672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:53.152764    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 20:09:53.375403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:53.373902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:54.130337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:54.130298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:54.151631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:54.151598    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:09:54.151847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:54.151697    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 20:09:54.746829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:54.746784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:55.156756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:55.156005    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:09:55.156756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:55.156064    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 20:09:55.341170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:55.341127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:55.936027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:55.935984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:56.152161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:56.152105    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:09:56.152445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:56.152407    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 20:09:56.344821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:56.344791    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:09:56.550935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:56.550893    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:57.154491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:57.154449    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:57.156407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:57.156381    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:57.156759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:57.156732    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:57.157492 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:57.157471    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:09:57.370745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:57.370713    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:57.937678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:57.934291    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:58.746944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:58.746909    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:09:59.170378 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:09:59.170285    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:09:59.465948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:09:59.465916    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:00.162432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:00.162385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:00.951459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:00.951415    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:01.674302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:01.674257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:02.322536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:02.322501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:03.190342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:03.190311    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:04.880672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:04.869775    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:10:04.920532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:04.920489    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:04.955772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:04.955730    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:05.035230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.035190    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"]
Nov 01 20:10:05.040411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.040383    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.042778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.042751    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"]
Nov 01 20:10:05.050166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.050056    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"distinct-2-w0-3a6480-kserve-self-signed-certs\""
Nov 01 20:10:05.070899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.070861    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:05.086679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.086629    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zfr\" (UniqueName: \"kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.086824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.086803    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.086945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.086926    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.087012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.086964    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.087103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.087082    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.087165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.087122    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.152415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.152327    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:05.188061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.188019    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.188246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.188069    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.188246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.188135    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.188246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.188164    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.188246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.188230    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zfr\" (UniqueName: \"kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.188407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.188366    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.189113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.189088    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.189286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.189120    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.189418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.189290    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.191294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.191269    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.191431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.191409    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.196447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.196419    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zfr\" (UniqueName: \"kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr\") pod \"distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.365585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.365552    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:05.659697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.659668    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"]
Nov 01 20:10:05.662488 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:10:05.662457    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf54274d3_3b0d_434f_8da7_6457847570fc.slice/crio-afa7f1cc2eab98eb9fc357262b8ea17580859bb96b17a690306a05bb25be7735 WatchSource:0}: Error finding container afa7f1cc2eab98eb9fc357262b8ea17580859bb96b17a690306a05bb25be7735: Status 404 returned error can't find the container with id afa7f1cc2eab98eb9fc357262b8ea17580859bb96b17a690306a05bb25be7735
Nov 01 20:10:05.938896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:05.938856    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:06.607296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:06.607263    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" event={"ID":"f54274d3-3b0d-434f-8da7-6457847570fc","Type":"ContainerStarted","Data":"bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f"}
Nov 01 20:10:06.607490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:06.607307    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" event={"ID":"f54274d3-3b0d-434f-8da7-6457847570fc","Type":"ContainerStarted","Data":"afa7f1cc2eab98eb9fc357262b8ea17580859bb96b17a690306a05bb25be7735"}
Nov 01 20:10:06.628032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:06.627983    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podStartSLOduration=1.627967893 podStartE2EDuration="1.627967893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:06.625695212 +0000 UTC m=+652.041327685" watchObservedRunningTime="2026-11-01 20:10:06.627967893 +0000 UTC m=+652.043600366"
Nov 01 20:10:06.667426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:06.667395    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:07.365969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:07.365937    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:07.379238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:07.379181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:07.971240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:07.971208    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:08.644708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:08.644679    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:09.271427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:09.271387    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"]
Nov 01 20:10:09.291131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:09.291091    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:09.393411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:09.393375    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:09.794451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:09.794401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:10.128045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:10.127952    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:10.746939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:10.746908    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:11.355256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:11.355218    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:12.130235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:12.130188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:12.734134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:12.734090    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:13.415333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:13.415283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:13.621681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:13.620332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:14.171252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:14.171217    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:10:14.414419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:14.414388    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:15.375670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:15.374749    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:15.794029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:15.793995    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:15.884193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:15.884156    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:15.969001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:15.968968    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:16.021866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:16.021823    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"]
Nov 01 20:10:16.167792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:16.167681    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:16.175356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:16.175140    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"]
Nov 01 20:10:16.345701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:16.344416    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:17.568657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:17.568616    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:18.243830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:18.243795    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:18.779601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:18.779559    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:19.098067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:19.096026    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:19.850717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:19.850683    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:20.635270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:20.635233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:21.022690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:21.022613    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" podUID="b072add5-cfbc-480e-a954-df9adeae1686" containerName="payload-processing" containerID="cri-o://88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6" gracePeriod=25
Nov 01 20:10:21.175764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:21.175721    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerName="payload-pre-processing" containerID="cri-o://7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889" gracePeriod=25
Nov 01 20:10:21.311621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:21.311530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:21.484328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:21.484294    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:21.966977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:21.966947    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:22.116860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:22.116825    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:22.234405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:22.234320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:22.608911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:22.608819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:22.936506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:22.936424    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:23.039296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:23.039260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:23.649149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:23.649114    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:23.751672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:23.751141    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:24.293555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.293509    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main" containerID="cri-o://265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e" gracePeriod=45
Nov 01 20:10:24.352619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.352577    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:24.639232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.639205    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:10:24.642989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.642965    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs\" (UniqueName: \"kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs\") pod \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") "
Nov 01 20:10:24.643137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643119    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir\") pod \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") "
Nov 01 20:10:24.643277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643144    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm\") pod \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") "
Nov 01 20:10:24.643277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643184    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms\" (UniqueName: \"kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms\") pod \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") "
Nov 01 20:10:24.643277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643215    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home\") pod \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") "
Nov 01 20:10:24.643277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643247    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache\") pod \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\" (UID: \"9486b112-3b6b-4bbf-9cb8-b032c585eb63\") "
Nov 01 20:10:24.643512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643328    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir" pod "9486b112-3b6b-4bbf-9cb8-b032c585eb63" (UID: "9486b112-3b6b-4bbf-9cb8-b032c585eb63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:24.643512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643478    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home" pod "9486b112-3b6b-4bbf-9cb8-b032c585eb63" (UID: "9486b112-3b6b-4bbf-9cb8-b032c585eb63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:24.643695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643675    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache" pod "9486b112-3b6b-4bbf-9cb8-b032c585eb63" (UID: "9486b112-3b6b-4bbf-9cb8-b032c585eb63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:24.643894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643798    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:24.643894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643818    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:24.643894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.643832    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:24.645408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.645381    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm" pod "9486b112-3b6b-4bbf-9cb8-b032c585eb63" (UID: "9486b112-3b6b-4bbf-9cb8-b032c585eb63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:24.645591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.645562    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs" pod "9486b112-3b6b-4bbf-9cb8-b032c585eb63" (UID: "9486b112-3b6b-4bbf-9cb8-b032c585eb63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:24.645663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.645605    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms" pod "9486b112-3b6b-4bbf-9cb8-b032c585eb63" (UID: "9486b112-3b6b-4bbf-9cb8-b032c585eb63"). InnerVolumeSpecName "kube-api-access-4sgms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:24.745556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.745515    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9486b112-3b6b-4bbf-9cb8-b032c585eb63-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:24.746007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.745694    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9486b112-3b6b-4bbf-9cb8-b032c585eb63-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:24.746007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.745718    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sgms\" (UniqueName: \"kubernetes.io/projected/9486b112-3b6b-4bbf-9cb8-b032c585eb63-kube-api-access-4sgms\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:24.836022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.835980    6479 generic.go:356] "Generic (PLEG): container finished" podID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerID="265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e" exitCode=0
Nov 01 20:10:24.836201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.836029    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"
Nov 01 20:10:24.836201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.836061    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" event={"ID":"9486b112-3b6b-4bbf-9cb8-b032c585eb63","Type":"ContainerDied","Data":"265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e"}
Nov 01 20:10:24.836201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.836107    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf" event={"ID":"9486b112-3b6b-4bbf-9cb8-b032c585eb63","Type":"ContainerDied","Data":"eceb3ae6dc296d0361bba3720681370cd946e643a53f46e2f5343b8a0976d5f5"}
Nov 01 20:10:24.836201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.836125    6479 scope.go:122] "RemoveContainer" containerID="265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e"
Nov 01 20:10:24.845505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.845483    6479 scope.go:122] "RemoveContainer" containerID="265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e"
Nov 01 20:10:24.845846 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:24.845824    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e\": container with ID starting with 265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e not found: ID does not exist" containerID="265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e"
Nov 01 20:10:24.845906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.845855    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e"} err="failed to get container status \"265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e\": rpc error: code = NotFound desc = could not find container \"265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e\": container with ID starting with 265917272a6858e22116a898278bb47bd28b98e03ebdf582b359204ac7f2bb6e not found: ID does not exist"
Nov 01 20:10:24.851741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.851708    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"]
Nov 01 20:10:24.855068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:24.855040    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4a127b/test-model-4a127b-kserve-7c4fb87f7c-z59gf"]
Nov 01 20:10:25.146609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:25.146577    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:25.956498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:25.956465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:26.307532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.307456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:26.337235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.337203    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"]
Nov 01 20:10:26.337840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.337820    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main"
Nov 01 20:10:26.337840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.337840    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main"
Nov 01 20:10:26.338032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.338019    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" containerName="main"
Nov 01 20:10:26.343093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.343073    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.353588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.353563    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"]
Nov 01 20:10:26.400681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.387548    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.400681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.387633    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.400681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.387680    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.400681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.387719    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.400681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.387760    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.426670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.423918    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnkw\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.426670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.424017    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.426670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.424136    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.426670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.424240    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.536824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.528039    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.538774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.537983    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.539316 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.538785    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.539519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.539499    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.539717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.539700    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.539842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.539827    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.539931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.539919    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.540058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.540045    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.540201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.540177    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnkw\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.540343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.540329    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.540903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.540881    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.541272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.541254    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.541626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.541603    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.541986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.541968    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.553799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.553767    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.555014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.554674    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.555141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.555054    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.557285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.557259    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnkw\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.671992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.671905    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:26.708470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.708438    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9486b112-3b6b-4bbf-9cb8-b032c585eb63" path="/var/lib/kubelet/pods/9486b112-3b6b-4bbf-9cb8-b032c585eb63/volumes"
Nov 01 20:10:26.747440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.747372    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:26.961778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.961714    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"]
Nov 01 20:10:26.964453 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:10:26.964419    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aac7bb0_b360_44df_b3a7_7a57d4eee4f0.slice/crio-a5716a76bbd5a86b490f007d5216bd6a43b1f15d72bde552f102be8f01fedc25 WatchSource:0}: Error finding container a5716a76bbd5a86b490f007d5216bd6a43b1f15d72bde552f102be8f01fedc25: Status 404 returned error can't find the container with id a5716a76bbd5a86b490f007d5216bd6a43b1f15d72bde552f102be8f01fedc25
Nov 01 20:10:26.966913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.966881    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:10:26.967011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.966976    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:10:26.967011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:26.967008    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:10:27.647412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:27.647374    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:27.925976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:27.925855    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" event={"ID":"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0","Type":"ContainerStarted","Data":"29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5"}
Nov 01 20:10:27.926260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:27.926220    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" event={"ID":"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0","Type":"ContainerStarted","Data":"a5716a76bbd5a86b490f007d5216bd6a43b1f15d72bde552f102be8f01fedc25"}
Nov 01 20:10:27.999918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:27.999846    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podStartSLOduration=1.999818863 podStartE2EDuration="1.999818863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:27.98825189 +0000 UTC m=+673.403884364" watchObservedRunningTime="2026-11-01 20:10:27.999818863 +0000 UTC m=+673.415451375"
Nov 01 20:10:28.583669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.582746    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:28.584015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.583942    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" podUID="b072add5-cfbc-480e-a954-df9adeae1686" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:28.723671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.723446    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:28.765443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.765399    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:28.784983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.784934    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:28.865729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.862737    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:28.915041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.914997    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:29.006669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:28.996744    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:29.318386 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:29.318319    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:10:29.430859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.428871    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:29.672737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.672035    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"]
Nov 01 20:10:29.673350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.673308    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:29.673553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.673523    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:29.746318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.745633    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"]
Nov 01 20:10:29.812901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.811853    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:29.988745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.967992    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:10:29.988745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.968771    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:10:29.988745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:29.970160    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:30.023844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.000630    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:10:30.023844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.001386    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:10:30.023844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.022803    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:30.259768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.237012    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:30.681586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.681487    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:30.681948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.681852    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:30.858232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.858175    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:30.859686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:30.859355    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:31.065672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.029240    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:31.084439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.056625    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:31.084439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.056708    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:10:31.084439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.069597    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:10:31.084439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.084144    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:10:31.674268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.674235    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:31.675144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.674540    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:31.858836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:31.858793    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:32.066103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.065759    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:32.081771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.081016    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:32.100407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.098728    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:32.100407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.099061    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:10:32.589071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.589027    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:10:32.650662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.636873    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:10:32.650662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.638284    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:32.650662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.646710    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:10:32.688621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.688173    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:32.688621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.688234    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:32.756989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:32.756956    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:33.287779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.287745    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"]
Nov 01 20:10:33.288583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.288545    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt" podUID="acfbe007-29f2-47c0-bf77-db7325485d77" containerName="maas-api" containerID="cri-o://5ca17e9dd3cefa84d4f784b6127aaa1ba47901efb8d5d2d3e19b32e9b350e84a" gracePeriod=30
Nov 01 20:10:33.311196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.311164    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:33.492026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.491980    6479 generic.go:356] "Generic (PLEG): container finished" podID="acfbe007-29f2-47c0-bf77-db7325485d77" containerID="5ca17e9dd3cefa84d4f784b6127aaa1ba47901efb8d5d2d3e19b32e9b350e84a" exitCode=0
Nov 01 20:10:33.492306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.492244    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt" event={"ID":"acfbe007-29f2-47c0-bf77-db7325485d77","Type":"ContainerDied","Data":"5ca17e9dd3cefa84d4f784b6127aaa1ba47901efb8d5d2d3e19b32e9b350e84a"}
Nov 01 20:10:33.585433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.585295    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:33.688660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.688228    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:33.688660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.688287    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:33.764688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.746667    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:33.938544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:33.938516    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:10:34.066057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.066021    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca\") pod \"acfbe007-29f2-47c0-bf77-db7325485d77\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") "
Nov 01 20:10:34.084679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.071577    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs\") pod \"acfbe007-29f2-47c0-bf77-db7325485d77\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") "
Nov 01 20:10:34.085864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.085833    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls\") pod \"acfbe007-29f2-47c0-bf77-db7325485d77\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") "
Nov 01 20:10:34.086190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.086170    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm\" (UniqueName: \"kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm\") pod \"acfbe007-29f2-47c0-bf77-db7325485d77\" (UID: \"acfbe007-29f2-47c0-bf77-db7325485d77\") "
Nov 01 20:10:34.087221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.070855    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca" pod "acfbe007-29f2-47c0-bf77-db7325485d77" (UID: "acfbe007-29f2-47c0-bf77-db7325485d77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:34.100066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.100042    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls" pod "acfbe007-29f2-47c0-bf77-db7325485d77" (UID: "acfbe007-29f2-47c0-bf77-db7325485d77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:34.100209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.100064    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs" pod "acfbe007-29f2-47c0-bf77-db7325485d77" (UID: "acfbe007-29f2-47c0-bf77-db7325485d77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:34.101297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.101271    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm" pod "acfbe007-29f2-47c0-bf77-db7325485d77" (UID: "acfbe007-29f2-47c0-bf77-db7325485d77"). InnerVolumeSpecName "kube-api-access-ckhdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:34.188709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.188558    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acfbe007-29f2-47c0-bf77-db7325485d77-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:34.188709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.188597    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:34.188709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.188610    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acfbe007-29f2-47c0-bf77-db7325485d77-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:34.188709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.188622    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhdm\" (UniqueName: \"kubernetes.io/projected/acfbe007-29f2-47c0-bf77-db7325485d77-kube-api-access-ckhdm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:34.425661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.425625    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:34.522810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.522769    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt" event={"ID":"acfbe007-29f2-47c0-bf77-db7325485d77","Type":"ContainerDied","Data":"ac96c6cf49239d54447f87a95eebb9404c396f292626c4ca18377c8becaeb2a9"}
Nov 01 20:10:34.523104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.523085    6479 scope.go:122] "RemoveContainer" containerID="5ca17e9dd3cefa84d4f784b6127aaa1ba47901efb8d5d2d3e19b32e9b350e84a"
Nov 01 20:10:34.523370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.523349    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"
Nov 01 20:10:34.564801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.564767    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"]
Nov 01 20:10:34.580350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.580313    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a127b-b4c98c9c9-prhxt"]
Nov 01 20:10:34.672760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.672716    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerName="payload-processing" containerID="cri-o://5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae" gracePeriod=25
Nov 01 20:10:34.674045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.674017    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:34.674266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.674236    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:34.709904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.709875    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfbe007-29f2-47c0-bf77-db7325485d77" path="/var/lib/kubelet/pods/acfbe007-29f2-47c0-bf77-db7325485d77/volumes"
Nov 01 20:10:34.745971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:34.745937    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerName="payload-pre-processing" containerID="cri-o://02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c" gracePeriod=25
Nov 01 20:10:35.171670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:35.168148    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:35.673667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:35.673226    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 20:10:35.673667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:35.673285    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 20:10:35.731294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:35.731256    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:35.938164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:35.938063    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:36.297168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:36.297133    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:36.606164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:36.606122    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:36.882671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:36.882563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:37.677669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.676979    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:37.677669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.677499    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:37.678957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.678785    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:37.709505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.709472    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:10:37.709718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.709520    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:37.842978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.842939    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:37.843539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.843515    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:37.844316 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:37.844296    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:10:38.012144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.012111    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:38.161604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.161562    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:38.262125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.262091    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:38.615482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.614861    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:38.640696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.634384    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" podUID="b072add5-cfbc-480e-a954-df9adeae1686" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:38.658167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.655197    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"]
Nov 01 20:10:38.658167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.656083    6479 cpu_manager.go:414] "containerMap: removing container" podUID="acfbe007-29f2-47c0-bf77-db7325485d77" containerName="maas-api"
Nov 01 20:10:38.658167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.656156    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfbe007-29f2-47c0-bf77-db7325485d77" containerName="maas-api"
Nov 01 20:10:38.658167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.656315    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfbe007-29f2-47c0-bf77-db7325485d77" containerName="maas-api"
Nov 01 20:10:38.706438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.692941    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.706438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.698129    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"]
Nov 01 20:10:38.743459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.741948    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-638f19-openshift-default-dockercfg-k2znv\""
Nov 01 20:10:38.743459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.743373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:38.805605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.805557    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.806031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.806008    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.806261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.806244    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.806431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.806415    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.806589 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.806575    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6td\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.806761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.806744    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.806908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.806893    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.807071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.807056    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.807247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.807233    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.823189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.823147    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"]
Nov 01 20:10:38.828571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.828547    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:38.836185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.836081    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-0d951a30aed7\""
Nov 01 20:10:38.840878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.839412    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"]
Nov 01 20:10:38.851384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.851357    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:38.915688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915326    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.915688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915537    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.915688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915587    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6td\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.915688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915627    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.916047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915758    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.916047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915801    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.916047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915849    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:38.916047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915914    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.916047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915944    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vc5\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:38.916047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.915998    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.916346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.916077    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:38.916346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.916110    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:38.916346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.916150    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.942707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.917622    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.942707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.938397    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.950560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.950515    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.960973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.954350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:38.960973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.954815    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.966313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.965816    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6td\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.971814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.971782    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.987721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.986568    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.993242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.993211    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:38.998502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:38.998445    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:39.017319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.017291    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.017774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.017336    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.017774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.017614    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.017774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.017697    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vc5\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.018536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.018508    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.019387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.019137    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.021878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.021835    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.027749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.027715    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vc5\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.095707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.095002    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:39.151284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.151246    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.411464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.411426    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"]
Nov 01 20:10:39.412641 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:10:39.412613    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d10b4_c73c_461e_a9ac_0c17bb03fc3f.slice/crio-2919cebfbff3c2c3b697cba88c081d811cc14a1e0318ced7f774021a6289b977 WatchSource:0}: Error finding container 2919cebfbff3c2c3b697cba88c081d811cc14a1e0318ced7f774021a6289b977: Status 404 returned error can't find the container with id 2919cebfbff3c2c3b697cba88c081d811cc14a1e0318ced7f774021a6289b977
Nov 01 20:10:39.415123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.415092    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:10:39.415178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.415159    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:10:39.415221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.415188    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:10:39.445877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.445845    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"]
Nov 01 20:10:39.447921 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:10:39.447898    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b80844_b954_4f7b_b9d0_574164979e0a.slice/crio-f41951b0d9824aa86121138544adf0a094722a8c91660aa6bfdaf629a291058b WatchSource:0}: Error finding container f41951b0d9824aa86121138544adf0a094722a8c91660aa6bfdaf629a291058b: Status 404 returned error can't find the container with id f41951b0d9824aa86121138544adf0a094722a8c91660aa6bfdaf629a291058b
Nov 01 20:10:39.594674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.592524    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm" event={"ID":"d6b80844-b954-4f7b-b9d0-574164979e0a","Type":"ContainerStarted","Data":"0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344"}
Nov 01 20:10:39.594674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.592604    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm" event={"ID":"d6b80844-b954-4f7b-b9d0-574164979e0a","Type":"ContainerStarted","Data":"f41951b0d9824aa86121138544adf0a094722a8c91660aa6bfdaf629a291058b"}
Nov 01 20:10:39.594674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.592845    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:39.595800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.595769    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" event={"ID":"405d10b4-c73c-461e-a9ac-0c17bb03fc3f","Type":"ContainerStarted","Data":"9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"}
Nov 01 20:10:39.595936 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.595815    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" event={"ID":"405d10b4-c73c-461e-a9ac-0c17bb03fc3f","Type":"ContainerStarted","Data":"2919cebfbff3c2c3b697cba88c081d811cc14a1e0318ced7f774021a6289b977"}
Nov 01 20:10:39.608357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.608309    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm" podStartSLOduration=1.608294149 podStartE2EDuration="1.608294149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:39.607726128 +0000 UTC m=+685.023358601" watchObservedRunningTime="2026-11-01 20:10:39.608294149 +0000 UTC m=+685.023926623"
Nov 01 20:10:39.633322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.633274    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podStartSLOduration=1.633259599 podStartE2EDuration="1.633259599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:39.630738801 +0000 UTC m=+685.046371275" watchObservedRunningTime="2026-11-01 20:10:39.633259599 +0000 UTC m=+685.048892072"
Nov 01 20:10:39.663912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:39.663875    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:40.096002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:40.095965    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:40.102169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:40.102142    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:40.361156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:40.361062    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:40.605089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:40.605061    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:40.605994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:40.605972    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:10:41.245217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.245180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:41.381688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.379825    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:10:41.486973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.486421    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7\" (UniqueName: \"kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7\") pod \"b072add5-cfbc-480e-a954-df9adeae1686\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") "
Nov 01 20:10:41.486973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.486545    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume\") pod \"b072add5-cfbc-480e-a954-df9adeae1686\" (UID: \"b072add5-cfbc-480e-a954-df9adeae1686\") "
Nov 01 20:10:41.487732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.487576    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume" pod "b072add5-cfbc-480e-a954-df9adeae1686" (UID: "b072add5-cfbc-480e-a954-df9adeae1686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:41.491530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.491489    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7" pod "b072add5-cfbc-480e-a954-df9adeae1686" (UID: "b072add5-cfbc-480e-a954-df9adeae1686"). InnerVolumeSpecName "kube-api-access-vt4p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:41.539288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.539264    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:10:41.588358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.588319    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4p7\" (UniqueName: \"kubernetes.io/projected/b072add5-cfbc-480e-a954-df9adeae1686-kube-api-access-vt4p7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:41.588358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.588360    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072add5-cfbc-480e-a954-df9adeae1686-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:41.612213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.612179    6479 generic.go:356] "Generic (PLEG): container finished" podID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerID="7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889" exitCode=0
Nov 01 20:10:41.612380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.612231    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"
Nov 01 20:10:41.612380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.612261    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" event={"ID":"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d","Type":"ContainerDied","Data":"7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889"}
Nov 01 20:10:41.612380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.612325    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m" event={"ID":"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d","Type":"ContainerDied","Data":"12f7440d76bd24dda9eed3f35c3c653f55e26d4141066a143cbde5c7aaaa1841"}
Nov 01 20:10:41.612380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.612345    6479 scope.go:122] "RemoveContainer" containerID="7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889"
Nov 01 20:10:41.613694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.613638    6479 generic.go:356] "Generic (PLEG): container finished" podID="b072add5-cfbc-480e-a954-df9adeae1686" containerID="88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6" exitCode=0
Nov 01 20:10:41.613819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.613703    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"
Nov 01 20:10:41.613819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.613747    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" event={"ID":"b072add5-cfbc-480e-a954-df9adeae1686","Type":"ContainerDied","Data":"88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6"}
Nov 01 20:10:41.613819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.613779    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4" event={"ID":"b072add5-cfbc-480e-a954-df9adeae1686","Type":"ContainerDied","Data":"dd85de24ee67903f8e535859d2fecffe95b347929eb32d541806005e355b485a"}
Nov 01 20:10:41.623344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.623324    6479 scope.go:122] "RemoveContainer" containerID="7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889"
Nov 01 20:10:41.623634 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:41.623614    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889\": container with ID starting with 7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889 not found: ID does not exist" containerID="7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889"
Nov 01 20:10:41.623746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.623718    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889"} err="failed to get container status \"7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889\": rpc error: code = NotFound desc = could not find container \"7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889\": container with ID starting with 7f53fa9af1b4c557933d2da0a7cb2ea7d83bbaa448c431de5ff756e548da1889 not found: ID does not exist"
Nov 01 20:10:41.623814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.623747    6479 scope.go:122] "RemoveContainer" containerID="88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6"
Nov 01 20:10:41.632033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.632018    6479 scope.go:122] "RemoveContainer" containerID="88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6"
Nov 01 20:10:41.632278 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:41.632260    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6\": container with ID starting with 88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6 not found: ID does not exist" containerID="88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6"
Nov 01 20:10:41.632339 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.632287    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6"} err="failed to get container status \"88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6\": rpc error: code = NotFound desc = could not find container \"88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6\": container with ID starting with 88916efbae40fd0ead6f2873a5990bbb8aecdf96c7aa475bd410a54d737d72b6 not found: ID does not exist"
Nov 01 20:10:41.637463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.637444    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"]
Nov 01 20:10:41.639759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.639739    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3c4a7-574b6d4b8c-m86t4"]
Nov 01 20:10:41.689761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.689733    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6\" (UniqueName: \"kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6\") pod \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") "
Nov 01 20:10:41.689884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.689811    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume\") pod \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\" (UID: \"7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d\") "
Nov 01 20:10:41.690193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.690172    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume" pod "7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" (UID: "7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:41.690319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.690303    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:41.691817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.691789    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6" pod "7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" (UID: "7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d"). InnerVolumeSpecName "kube-api-access-98kw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:41.792824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.792747    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kw6\" (UniqueName: \"kubernetes.io/projected/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d-kube-api-access-98kw6\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:41.836888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.836857    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:10:41.864235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.864207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:41.937760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.937731    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"]
Nov 01 20:10:41.950829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:41.950794    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3c4a7-84d76cbb47-b8z7m"]
Nov 01 20:10:42.003895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:42.003859    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:42.153440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:42.153402    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:42.726708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:42.726629    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" path="/var/lib/kubelet/pods/7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d/volumes"
Nov 01 20:10:42.727865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:42.727842    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b072add5-cfbc-480e-a954-df9adeae1686" path="/var/lib/kubelet/pods/b072add5-cfbc-480e-a954-df9adeae1686/volumes"
Nov 01 20:10:43.226669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.226205    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:43.493278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.493189    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"]
Nov 01 20:10:43.525712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.494242    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerName="payload-pre-processing"
Nov 01 20:10:43.533738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.533703    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerName="payload-pre-processing"
Nov 01 20:10:43.534056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.534039    6479 cpu_manager.go:414] "containerMap: removing container" podUID="b072add5-cfbc-480e-a954-df9adeae1686" containerName="payload-processing"
Nov 01 20:10:43.534191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.534174    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b072add5-cfbc-480e-a954-df9adeae1686" containerName="payload-processing"
Nov 01 20:10:43.539833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.539800    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="b072add5-cfbc-480e-a954-df9adeae1686" containerName="payload-processing"
Nov 01 20:10:43.539997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.539982    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbd8abe-4cc0-4368-b6f2-cc4ea3e8062d" containerName="payload-pre-processing"
Nov 01 20:10:43.678667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.678394    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:43.681540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.681512    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"]
Nov 01 20:10:43.681900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.681872    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:43.827760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.824742    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-638f19-dockercfg-rn7jm\""
Nov 01 20:10:43.827760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.824842    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-638f19\""
Nov 01 20:10:43.904671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.865749    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"]
Nov 01 20:10:43.958663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.938294    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:43.958663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.889229    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:43.988670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.968261    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmjb\" (UniqueName: \"kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb\") pod \"payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:43.988670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:43.979194    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.009973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.004721    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"]
Nov 01 20:10:44.073212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.071914    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmjb\" (UniqueName: \"kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb\") pod \"payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:44.073212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.072512    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:44.073820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.073638    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:44.147722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.139100    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmjb\" (UniqueName: \"kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb\") pod \"payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:44.241671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.228512    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv456\" (UniqueName: \"kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456\") pod \"payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.241671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.228634    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.333730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.333422    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv456\" (UniqueName: \"kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456\") pod \"payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.333730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.333521    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.339669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.334459    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.368668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.357400    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv456\" (UniqueName: \"kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456\") pod \"payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.390196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.390169    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:44.451995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.451960    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:44.669672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.665076    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:44.998998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:44.998947    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:10:45.135870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:45.135827    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.136813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:45.136725    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.141745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:45.141715    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.666763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:45.666716    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.667768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:45.667746    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:45.669518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:45.669493    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:10:46.000080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.000034    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:46.008609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.008580    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:10:46.226778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.157745    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:46.226778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.158218    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:46.226778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.159023    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:10:46.226778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.159104    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:10:46.793686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.784033    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:46.927213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.926767    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 01 20:10:46.927213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:46.926871    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:10:47.076921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.076825    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:47.406665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.391918    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:10:47.406987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.406960    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:10:47.407122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.407109    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"]
Nov 01 20:10:47.436722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.407283    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:10:47.436722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.429017    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:10:47.436722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.435528    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:10:47.503596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.503488    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:10:47.552875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.552841    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"]
Nov 01 20:10:47.671329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.671193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:47.678736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.678316    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:47.679744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:47.679697    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:10:48.235200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.235097    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" event={"ID":"7ee6f741-9e37-48bc-a647-0620e3cd2362","Type":"ContainerStarted","Data":"d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b"}
Nov 01 20:10:48.235200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.235191    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" event={"ID":"7ee6f741-9e37-48bc-a647-0620e3cd2362","Type":"ContainerStarted","Data":"3c3d6d77c923c4d437b1be8e6f0b359d13829e4f7845ae9c11f4981b92d4e822"}
Nov 01 20:10:48.235734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.235298    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:48.236906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.236881    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" event={"ID":"8347908c-6d47-4167-9621-462e8f9c4d00","Type":"ContainerStarted","Data":"2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253"}
Nov 01 20:10:48.237011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.236921    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" event={"ID":"8347908c-6d47-4167-9621-462e8f9c4d00","Type":"ContainerStarted","Data":"dc2dd73b56f1c024bd0bf97b3e05e6f2c179f9e38a1e266fbf5f517ae67edd11"}
Nov 01 20:10:48.237271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.237251    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:48.252856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.252809    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" podStartSLOduration=5.252795865 podStartE2EDuration="5.252795865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:48.249973106 +0000 UTC m=+693.665605576" watchObservedRunningTime="2026-11-01 20:10:48.252795865 +0000 UTC m=+693.668428337"
Nov 01 20:10:48.457384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:48.457345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:49.455388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:49.455349    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:50.104179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.104127    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" podStartSLOduration=7.104112083 podStartE2EDuration="7.104112083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:48.265056863 +0000 UTC m=+693.680689337" watchObservedRunningTime="2026-11-01 20:10:50.104112083 +0000 UTC m=+695.519744559"
Nov 01 20:10:50.106432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.106403    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"]
Nov 01 20:10:50.113102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.113078    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.118342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.118316    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-638f19-serving-cert\""
Nov 01 20:10:50.122109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.122087    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"]
Nov 01 20:10:50.160224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.160195    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:50.188932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.188898    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.189115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.188954    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.189115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.189104    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8ms\" (UniqueName: \"kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.189255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.189231    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.253500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.253463    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:50.303676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.303489    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.303676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.303621    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8ms\" (UniqueName: \"kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.303933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.303729    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.303996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.303958    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.307663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.304862    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.335671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.313927    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.345741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.344322    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.345741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.344671    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8ms\" (UniqueName: \"kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms\") pod \"maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.436270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.436192    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:50.608334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.608303    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:10:50.654856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.654821    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"]
Nov 01 20:10:50.722074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:50.721534    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"]
Nov 01 20:10:50.725595 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:10:50.724574    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e427c6_3721_46de_bd53_58ff20a46d5b.slice/crio-2881d9207c6a85bdce42b7f6c8659e9ad17f32bf3de535f88a2d62ac2b1e77a2 WatchSource:0}: Error finding container 2881d9207c6a85bdce42b7f6c8659e9ad17f32bf3de535f88a2d62ac2b1e77a2: Status 404 returned error can't find the container with id 2881d9207c6a85bdce42b7f6c8659e9ad17f32bf3de535f88a2d62ac2b1e77a2
Nov 01 20:10:51.038334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:51.038233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:51.257079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:51.257047    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" event={"ID":"01e427c6-3721-46de-bd53-58ff20a46d5b","Type":"ContainerStarted","Data":"2881d9207c6a85bdce42b7f6c8659e9ad17f32bf3de535f88a2d62ac2b1e77a2"}
Nov 01 20:10:51.643952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:51.643850    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:52.245198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:52.245151    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:52.264871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:52.264834    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" event={"ID":"01e427c6-3721-46de-bd53-58ff20a46d5b","Type":"ContainerStarted","Data":"4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708"}
Nov 01 20:10:52.265049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:52.264941    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:52.282894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:52.282762    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" podStartSLOduration=1.769906166 podStartE2EDuration="2.28274637s" totalImagesPullingTime="512.840204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:52.281430185 +0000 UTC m=+697.697062659" watchObservedRunningTime="2026-11-01 20:10:52.28274637 +0000 UTC m=+697.698378849"
Nov 01 20:10:52.836347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:52.836310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:53.454246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:53.454192    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:54.049995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:54.049959    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:54.739288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:54.739261    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:55.091758 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.091735    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:10:55.096295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.096275    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:10:55.203384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.203290    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume\") pod \"057dad34-dbe0-4647-b35b-e350bb0f80d6\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") "
Nov 01 20:10:55.203384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.203360    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr\" (UniqueName: \"kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr\") pod \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") "
Nov 01 20:10:55.203635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.203468    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk\" (UniqueName: \"kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk\") pod \"057dad34-dbe0-4647-b35b-e350bb0f80d6\" (UID: \"057dad34-dbe0-4647-b35b-e350bb0f80d6\") "
Nov 01 20:10:55.203635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.203525    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume\") pod \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\" (UID: \"421010e1-3dec-4e2e-818c-3b42ed9a9f9f\") "
Nov 01 20:10:55.203798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.203737    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume" pod "057dad34-dbe0-4647-b35b-e350bb0f80d6" (UID: "057dad34-dbe0-4647-b35b-e350bb0f80d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:55.203956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.203934    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume" pod "421010e1-3dec-4e2e-818c-3b42ed9a9f9f" (UID: "421010e1-3dec-4e2e-818c-3b42ed9a9f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:55.204190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.204174    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057dad34-dbe0-4647-b35b-e350bb0f80d6-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:55.204239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.204192    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:55.205488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.205463    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk" pod "057dad34-dbe0-4647-b35b-e350bb0f80d6" (UID: "057dad34-dbe0-4647-b35b-e350bb0f80d6"). InnerVolumeSpecName "kube-api-access-l4sgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:55.205488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.205483    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr" pod "421010e1-3dec-4e2e-818c-3b42ed9a9f9f" (UID: "421010e1-3dec-4e2e-818c-3b42ed9a9f9f"). InnerVolumeSpecName "kube-api-access-dk8vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:55.301164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.301121    6479 generic.go:356] "Generic (PLEG): container finished" podID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerID="02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c" exitCode=0
Nov 01 20:10:55.301359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.301192    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"
Nov 01 20:10:55.301359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.301205    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" event={"ID":"421010e1-3dec-4e2e-818c-3b42ed9a9f9f","Type":"ContainerDied","Data":"02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c"}
Nov 01 20:10:55.301359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.301317    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5" event={"ID":"421010e1-3dec-4e2e-818c-3b42ed9a9f9f","Type":"ContainerDied","Data":"28f0dc65724e85169ee10a6fad31a395e07100b2540c81c00803bb40b6642759"}
Nov 01 20:10:55.301359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.301348    6479 scope.go:122] "RemoveContainer" containerID="02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c"
Nov 01 20:10:55.303084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.303058    6479 generic.go:356] "Generic (PLEG): container finished" podID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerID="5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae" exitCode=0
Nov 01 20:10:55.303218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.303115    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" event={"ID":"057dad34-dbe0-4647-b35b-e350bb0f80d6","Type":"ContainerDied","Data":"5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae"}
Nov 01 20:10:55.303218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.303122    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"
Nov 01 20:10:55.303218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.303153    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b" event={"ID":"057dad34-dbe0-4647-b35b-e350bb0f80d6","Type":"ContainerDied","Data":"e10b6be19fec60d8fd6d2c4d136f3787e542c5372b37eaf8b1b8232a944125cf"}
Nov 01 20:10:55.304848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.304826    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8vr\" (UniqueName: \"kubernetes.io/projected/421010e1-3dec-4e2e-818c-3b42ed9a9f9f-kube-api-access-dk8vr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:55.304848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.304846    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sgk\" (UniqueName: \"kubernetes.io/projected/057dad34-dbe0-4647-b35b-e350bb0f80d6-kube-api-access-l4sgk\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:55.314335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.314275    6479 scope.go:122] "RemoveContainer" containerID="02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c"
Nov 01 20:10:55.314642 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:55.314621    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c\": container with ID starting with 02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c not found: ID does not exist" containerID="02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c"
Nov 01 20:10:55.314774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.314753    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c"} err="failed to get container status \"02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c\": rpc error: code = NotFound desc = could not find container \"02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c\": container with ID starting with 02c5d0d2514e9c05137dbb9d4ed181f5facd7216b7e4d7ba31fd9aea8291fb5c not found: ID does not exist"
Nov 01 20:10:55.314828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.314776    6479 scope.go:122] "RemoveContainer" containerID="5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae"
Nov 01 20:10:55.319692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.319625    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:55.324098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.324081    6479 scope.go:122] "RemoveContainer" containerID="5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae"
Nov 01 20:10:55.324409 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:55.324388    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae\": container with ID starting with 5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae not found: ID does not exist" containerID="5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae"
Nov 01 20:10:55.324500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.324413    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae"} err="failed to get container status \"5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae\": rpc error: code = NotFound desc = could not find container \"5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae\": container with ID starting with 5cc6b99cabf98ebfadc1ecdf85b20349f8b50d4f904f2826ef05c7eb0e7dbeae not found: ID does not exist"
Nov 01 20:10:55.325527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.325507    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"]
Nov 01 20:10:55.329357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.329335    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a127b-9b6d68d75-bkgs5"]
Nov 01 20:10:55.339658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.339623    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"]
Nov 01 20:10:55.344229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.344208    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4a127b-5f46976497-2lp9b"]
Nov 01 20:10:55.655530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:55.655437    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n" podUID="5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" containerName="payload-processing" containerID="cri-o://0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316" gracePeriod=25
Nov 01 20:10:56.059532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.059498    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:56.085185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.085158    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:10:56.248412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.248337    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume\") pod \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") "
Nov 01 20:10:56.248898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.248429    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume\") pod \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") "
Nov 01 20:10:56.248898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.248497    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn\") pod \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") "
Nov 01 20:10:56.248898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.248609    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials\") pod \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\" (UID: \"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2\") "
Nov 01 20:10:56.248898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.248744    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume" pod "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" (UID: "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:56.249055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.248893    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume" pod "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" (UID: "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:56.249119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.249103    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:56.249153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.249124    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:56.250926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.250907    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials" pod "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" (UID: "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:56.251084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.251054    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn" pod "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" (UID: "5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2"). InnerVolumeSpecName "kube-api-access-4tkvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:56.312499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.312463    6479 generic.go:356] "Generic (PLEG): container finished" podID="5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" containerID="0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316" exitCode=0
Nov 01 20:10:56.312679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.312504    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n" event={"ID":"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2","Type":"ContainerDied","Data":"0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316"}
Nov 01 20:10:56.312679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.312515    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"
Nov 01 20:10:56.312679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.312556    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n" event={"ID":"5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2","Type":"ContainerDied","Data":"6f08b4c2c8eb4a4c6b77e7b6990fceeaaa19ff6081eab8adf1f1d12c5919ad2f"}
Nov 01 20:10:56.312679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.312584    6479 scope.go:122] "RemoveContainer" containerID="0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316"
Nov 01 20:10:56.323627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.323603    6479 scope.go:122] "RemoveContainer" containerID="0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316"
Nov 01 20:10:56.323930 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:56.323909    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316\": container with ID starting with 0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316 not found: ID does not exist" containerID="0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316"
Nov 01 20:10:56.324019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.323940    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316"} err="failed to get container status \"0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316\": rpc error: code = NotFound desc = could not find container \"0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316\": container with ID starting with 0c402bf334cbb89e04b87eb1aec71d001ba7c40d1f448fc4c6dafbee94ac3316 not found: ID does not exist"
Nov 01 20:10:56.334307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.334284    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"]
Nov 01 20:10:56.338003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.337976    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-77897p29n"]
Nov 01 20:10:56.350023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.349998    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tkvn\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-kube-api-access-4tkvn\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:56.350023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.350023    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:56.787677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.787611    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" path="/var/lib/kubelet/pods/057dad34-dbe0-4647-b35b-e350bb0f80d6/volumes"
Nov 01 20:10:56.796669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.792543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:10:56.851671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.850907    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" path="/var/lib/kubelet/pods/421010e1-3dec-4e2e-818c-3b42ed9a9f9f/volumes"
Nov 01 20:10:56.866671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:56.866330    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" path="/var/lib/kubelet/pods/5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2/volumes"
Nov 01 20:10:58.334663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:58.325818    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:10:58.509672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:58.505293    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:10:59.090672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.089743    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:10:59.208672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.202783    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:59.295670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.295097    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:10:59.310671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.296421    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:10:59.310671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.308567    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:59.529674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.522002    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:10:59.572713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.557124    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:10:59.746680 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:10:59.721886    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:10:59.799856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:10:59.799750    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:00.101851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.101746    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:00.103463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.103428    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:00.216674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.208761    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:00.216674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.210277    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:11:00.410850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.410745    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:11:00.416671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.412696    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:00.578098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.578054    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:11:00.831702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:00.812141    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:11:01.141722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.012922    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:01.141722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.112803    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:01.235672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.230821    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:11:01.404732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.380043    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:01.411795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.411747    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:01.414179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.414052    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:01.583671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:01.582793    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:11:02.060346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.056504    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded"
Nov 01 20:11:02.124052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.119970    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:02.168337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.168297    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:02.221470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.221321    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:02.274667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.274598    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:02.314977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.313881    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:11:02.567353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.567269    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:02.598567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.598520    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:02.743842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:02.743809    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:11:03.040511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:03.040481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:03.859485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:03.859438    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:04.841838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:04.841798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:05.672168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:05.672132    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:06.198096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.198060    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:06.297809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.297780    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:06.409252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.409219    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:06.522366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522285    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"]
Nov 01 20:11:06.522840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522826    6479 cpu_manager.go:414] "containerMap: removing container" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerName="payload-pre-processing"
Nov 01 20:11:06.522891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522842    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerName="payload-pre-processing"
Nov 01 20:11:06.522891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522853    6479 cpu_manager.go:414] "containerMap: removing container" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerName="payload-processing"
Nov 01 20:11:06.522891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522862    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerName="payload-processing"
Nov 01 20:11:06.522891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522890    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" containerName="payload-processing"
Nov 01 20:11:06.523030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522895    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" containerName="payload-processing"
Nov 01 20:11:06.523030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522989    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="057dad34-dbe0-4647-b35b-e350bb0f80d6" containerName="payload-processing"
Nov 01 20:11:06.523030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.522999    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="421010e1-3dec-4e2e-818c-3b42ed9a9f9f" containerName="payload-pre-processing"
Nov 01 20:11:06.523030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.523007    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7ffa7c-6b2a-4dec-84c0-90891a1fa5b2" containerName="payload-processing"
Nov 01 20:11:06.526513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.526497    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.534123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.534097    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"agc-cred-9be1c0e3b0a7\""
Nov 01 20:11:06.534479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.534457    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e38ef4\""
Nov 01 20:11:06.534618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.534597    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"routing-overlay\""
Nov 01 20:11:06.534797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.534610    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"openshift-service-ca.crt\""
Nov 01 20:11:06.534933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.534523    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"payload-processing-external-model-e2e--fb221618-dockercfg-vcdb4\""
Nov 01 20:11:06.535038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.534704    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"kube-root-ca.crt\""
Nov 01 20:11:06.538388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.538363    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"]
Nov 01 20:11:06.584952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.584911    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtxk\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.585147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.585053    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.585221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.585139    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.585308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.585293    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.625502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.625468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:06.642676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.642631    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"]
Nov 01 20:11:06.646542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.646518    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.655794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.655767    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"]
Nov 01 20:11:06.655960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.655944    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-cb46214ead48\""
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.686879    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.686933    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.686978    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.687020    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.687141    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.687190    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.687240    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4p7\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.687764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.687299    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtxk\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.691428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.688340    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.691428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.689955    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.691664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.691556    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:06.890995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.855574    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.890995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.855808    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.890995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.855864    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4p7\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.890995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.855993    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.890995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.857016    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.923537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.892498    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.923537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.901430    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:06.944280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.944200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:06.977337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:06.977284    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtxk\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:07.020411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:07.020367    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4p7\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:07.144107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:07.144016    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:07.264603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:07.264536    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:07.431878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:07.431847    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"]
Nov 01 20:11:07.433118 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:11:07.433077    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e440a1e_e917_493f_9b48_2cd5392d67ac.slice/crio-c85a15c2dc525a2d11a2563a3ecb53417a08b36fa07720b1bf565f5f74b65a36 WatchSource:0}: Error finding container c85a15c2dc525a2d11a2563a3ecb53417a08b36fa07720b1bf565f5f74b65a36: Status 404 returned error can't find the container with id c85a15c2dc525a2d11a2563a3ecb53417a08b36fa07720b1bf565f5f74b65a36
Nov 01 20:11:07.449928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:07.449902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:07.611118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:07.611087    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"]
Nov 01 20:11:08.062894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.062681    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4" event={"ID":"b4c3520b-1ca1-4844-a37b-dbcdbd062872","Type":"ContainerStarted","Data":"ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff"}
Nov 01 20:11:08.062894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.062748    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4" event={"ID":"b4c3520b-1ca1-4844-a37b-dbcdbd062872","Type":"ContainerStarted","Data":"1e05b488e32ae1805988d5888e3692b8c3a3e6432a690129a5c4992170c15360"}
Nov 01 20:11:08.063357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.062894    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:08.063904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.063845    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:08.065069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.065044    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" event={"ID":"9e440a1e-e917-493f-9b48-2cd5392d67ac","Type":"ContainerStarted","Data":"b3fb7da18ea3d5cd4af7ea851fb03d6fefad25b0c9bbb9d84c98b050a37127b4"}
Nov 01 20:11:08.065183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.065081    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" event={"ID":"9e440a1e-e917-493f-9b48-2cd5392d67ac","Type":"ContainerStarted","Data":"c85a15c2dc525a2d11a2563a3ecb53417a08b36fa07720b1bf565f5f74b65a36"}
Nov 01 20:11:08.065183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.065108    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:08.080953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.080886    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4" podStartSLOduration=2.080864917 podStartE2EDuration="2.080864917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:08.079134838 +0000 UTC m=+713.494767312" watchObservedRunningTime="2026-11-01 20:11:08.080864917 +0000 UTC m=+713.496497390"
Nov 01 20:11:08.099858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.099805    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" podStartSLOduration=2.099789738 podStartE2EDuration="2.099789738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:08.095982928 +0000 UTC m=+713.511615403" watchObservedRunningTime="2026-11-01 20:11:08.099789738 +0000 UTC m=+713.515422232"
Nov 01 20:11:08.247490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.247453    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:08.459675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.459609    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"]
Nov 01 20:11:08.465194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.465165    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.474298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.474255    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7f0c86-openshift-default-dockercfg-plw2l\""
Nov 01 20:11:08.476315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.476286    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"]
Nov 01 20:11:08.502626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.501062    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:08.573487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.573447    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"]
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601304    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601359    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601425    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtcd\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601508    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601549    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601711    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.601783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601774    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.602297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601799    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.602297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.601825    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.603602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.603570    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:08.687750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.687710    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:08.702709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702671    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.702905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702717    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.702905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702748    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.702905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702785    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.702905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702809    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.702905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702874    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtcd\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.702974    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.703020    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.703153    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.703184    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.703346    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.703445    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.703570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.703530    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.704207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.704177    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.706136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.706112    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.709126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.709096    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.710274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.710219    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.711129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.711104    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtcd\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd\") pod \"e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") " pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:08.801166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:08.800889    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:09.075178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.075096    6479 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" secret="" err="secret \"payload-processing-external-model-e2e--fb221618-dockercfg-vcdb4\" not found"
Nov 01 20:11:09.219958 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.219444    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:09.219958 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.219546    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-9be1c0e3b0a7: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:09.219958 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.219570    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:09.219958 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.219555    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:09.719532529 +0000 UTC m=+715.135165002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:09.219958 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.219858    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:09.719839847 +0000 UTC m=+715.135472310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:09.221974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.221866    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"]
Nov 01 20:11:09.224679 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:11:09.224638    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3ab07f_1da6_442f_857b_5789828beea8.slice/crio-954e9d8af83f516991f0ed61b7a125534c43eabc3ae8ce5f20b4c107cf9771a8 WatchSource:0}: Error finding container 954e9d8af83f516991f0ed61b7a125534c43eabc3ae8ce5f20b4c107cf9771a8: Status 404 returned error can't find the container with id 954e9d8af83f516991f0ed61b7a125534c43eabc3ae8ce5f20b4c107cf9771a8
Nov 01 20:11:09.227488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.227448    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:09.227588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.227529    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:09.227588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.227570    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:09.247236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.247204    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:09.752764 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.752731    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:09.752935 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.752761    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-9be1c0e3b0a7: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:09.752935 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.752831    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:10.752812416 +0000 UTC m=+716.168444871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:09.752935 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.752836    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:09.752935 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:09.752900    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:10.752882238 +0000 UTC m=+716.168514693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:09.854793 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:09.854756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:10.085861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.085767    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" event={"ID":"5b3ab07f-1da6-442f-857b-5789828beea8","Type":"ContainerStarted","Data":"10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"}
Nov 01 20:11:10.085861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.085837    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" event={"ID":"5b3ab07f-1da6-442f-857b-5789828beea8","Type":"ContainerStarted","Data":"954e9d8af83f516991f0ed61b7a125534c43eabc3ae8ce5f20b4c107cf9771a8"}
Nov 01 20:11:10.107395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.107334    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podStartSLOduration=2.107315115 podStartE2EDuration="2.107315115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:10.105708081 +0000 UTC m=+715.521340555" watchObservedRunningTime="2026-11-01 20:11:10.107315115 +0000 UTC m=+715.522947589"
Nov 01 20:11:10.548696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.548660    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:10.724788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.722814    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:39272->10.134.0.58:15021: read: connection reset by peer" start-of-body=
Nov 01 20:11:10.724788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.722887    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:39272->10.134.0.58:15021: read: connection reset by peer"
Nov 01 20:11:10.776485 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:10.776446    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-9be1c0e3b0a7: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:10.776485 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:10.776474    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:10.776751 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:10.776559    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:12.776536296 +0000 UTC m=+718.192168763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:10.776751 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:10.776476    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:10.776751 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:10.776611    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:12.776598833 +0000 UTC m=+718.192231301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:10.802125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.802038    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:10.807822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:10.807797    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:11.094696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.094583    6479 generic.go:356] "Generic (PLEG): container finished" podID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerID="9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c" exitCode=0
Nov 01 20:11:11.094696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.094631    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" event={"ID":"405d10b4-c73c-461e-a9ac-0c17bb03fc3f","Type":"ContainerDied","Data":"9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"}
Nov 01 20:11:11.095156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.095120    6479 scope.go:122] "RemoveContainer" containerID="9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"
Nov 01 20:11:11.095373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.095353    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:11.096176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.096151    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:11.096733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.096705    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:11.096819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.096775    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:11.096819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.096805    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:11.150957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.150919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:11.514254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.514212    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:11:11.955668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:11.954991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:12.128671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:12.128006    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" event={"ID":"405d10b4-c73c-461e-a9ac-0c17bb03fc3f","Type":"ContainerStarted","Data":"bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68"}
Nov 01 20:11:12.336696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:12.336551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:12.438116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:12.438067    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:12.881809 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:12.881776    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-9be1c0e3b0a7: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:12.882038 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:12.882026    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774: secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:12.882184 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:12.882173    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:16.882154245 +0000 UTC m=+722.297786711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : secret "agc-cred-9be1c0e3b0a7" not found
Nov 01 20:11:12.882708 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:12.882689    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:12.882877 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:12.882866    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume podName:9e440a1e-e917-493f-9b48-2cd5392d67ac nodeName:}" failed. No retries permitted until 2026-11-01 20:11:16.882849362 +0000 UTC m=+722.298481827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:13.102671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:13.098142    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:11:13.102671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:13.098280    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:13.102671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:13.098322    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:14.106072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.099111    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:14.106072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.099192    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:14.123671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.117772    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:14.199671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.178972    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:14.284671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.281797    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:14.429731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.428811    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:14.603672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:14.595755    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:15.091675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.087729    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" podUID="9e440a1e-e917-493f-9b48-2cd5392d67ac" containerName="payload-processing" containerID="cri-o://b3fb7da18ea3d5cd4af7ea851fb03d6fefad25b0c9bbb9d84c98b050a37127b4" gracePeriod=25
Nov 01 20:11:15.096217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.096186    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:15.096449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.096421    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:15.144943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.125793    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:15.144943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.126544    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:15.241698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.188094    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:15.241698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.188969    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:15.241698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.193334    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:15.241698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.236399    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:15.428763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.307194    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:11:15.428763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.342261    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:11:15.467952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.435948    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:11:15.467952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.436786    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:15.467952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.450625    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:15.774693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.744422    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:15.787909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.787868    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:11:15.912365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:15.893927    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:16.228366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.228335    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:16.238954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.238774    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:16.243814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.242774    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:16.243814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.242818    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:16.249015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.246801    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:11:16.333706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.329690    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:11:16.333706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.330072    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:16.395513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.361669    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:16.429407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.429368    6479 generic.go:356] "Generic (PLEG): container finished" podID="9e440a1e-e917-493f-9b48-2cd5392d67ac" containerID="b3fb7da18ea3d5cd4af7ea851fb03d6fefad25b0c9bbb9d84c98b050a37127b4" exitCode=0
Nov 01 20:11:16.429972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.429723    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" event={"ID":"9e440a1e-e917-493f-9b48-2cd5392d67ac","Type":"ContainerDied","Data":"b3fb7da18ea3d5cd4af7ea851fb03d6fefad25b0c9bbb9d84c98b050a37127b4"}
Nov 01 20:11:16.429972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.429789    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774" event={"ID":"9e440a1e-e917-493f-9b48-2cd5392d67ac","Type":"ContainerDied","Data":"c85a15c2dc525a2d11a2563a3ecb53417a08b36fa07720b1bf565f5f74b65a36"}
Nov 01 20:11:16.429972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.429813    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85a15c2dc525a2d11a2563a3ecb53417a08b36fa07720b1bf565f5f74b65a36"
Nov 01 20:11:16.434714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.434690    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:16.519331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.518753    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk\") pod \"9e440a1e-e917-493f-9b48-2cd5392d67ac\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") "
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.524335    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume\") pod \"9e440a1e-e917-493f-9b48-2cd5392d67ac\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") "
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.524429    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials\") pod \"9e440a1e-e917-493f-9b48-2cd5392d67ac\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") "
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.524667    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume\") pod \"9e440a1e-e917-493f-9b48-2cd5392d67ac\" (UID: \"9e440a1e-e917-493f-9b48-2cd5392d67ac\") "
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.524136    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk" pod "9e440a1e-e917-493f-9b48-2cd5392d67ac" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac"). InnerVolumeSpecName "kube-api-access-hvtxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.524994    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume" pod "9e440a1e-e917-493f-9b48-2cd5392d67ac" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.526390    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtxk\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-kube-api-access-hvtxk\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.526418    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.527975    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume" pod "9e440a1e-e917-493f-9b48-2cd5392d67ac" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:16.540718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.533952    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials" pod "9e440a1e-e917-493f-9b48-2cd5392d67ac" (UID: "9e440a1e-e917-493f-9b48-2cd5392d67ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:16.628302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.628261    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e440a1e-e917-493f-9b48-2cd5392d67ac-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:16.628523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.628507    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e440a1e-e917-493f-9b48-2cd5392d67ac-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:16.835693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:16.834549    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:17.096243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:17.096145    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:17.096243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:17.096213    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:17.434760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:17.434676    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"
Nov 01 20:11:17.474909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:17.474870    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"]
Nov 01 20:11:17.486862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:17.486706    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-76bblj774"]
Nov 01 20:11:17.572683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:17.572597    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:18.096232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:18.096192    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:18.096416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:18.096260    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:18.478194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:18.478156    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:18.709619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:18.709582    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e440a1e-e917-493f-9b48-2cd5392d67ac" path="/var/lib/kubelet/pods/9e440a1e-e917-493f-9b48-2cd5392d67ac/volumes"
Nov 01 20:11:19.081230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:19.081141    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:11:19.095739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:19.095708    6479 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 20:11:19.095890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:19.095768    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 20:11:19.130459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:19.130420    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"]
Nov 01 20:11:19.184674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:19.184538    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:19.816853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:19.816811    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:20.101183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.101102    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:11:20.101419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.101396    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:11:20.102061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.102043    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:11:20.589837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.589800    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"]
Nov 01 20:11:20.590694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.590640    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9e440a1e-e917-493f-9b48-2cd5392d67ac" containerName="payload-processing"
Nov 01 20:11:20.590878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.590862    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e440a1e-e917-493f-9b48-2cd5392d67ac" containerName="payload-processing"
Nov 01 20:11:20.591198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.591183    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e440a1e-e917-493f-9b48-2cd5392d67ac" containerName="payload-processing"
Nov 01 20:11:20.597139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.597115    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.604588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.603905    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"]
Nov 01 20:11:20.605927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.605764    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7f0c86-serving-cert\""
Nov 01 20:11:20.635253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.635218    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:20.734019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.733982    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.734186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.734136    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.734234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.734213    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x674s\" (UniqueName: \"kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.734371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.734350    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.736159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.736133    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:20.853738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.837019    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.853738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.837311    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.853738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.837433    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.853738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.837488    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x674s\" (UniqueName: \"kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.853738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.838830    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.858146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.858084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:20.859495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.859464    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.863680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.863417    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x674s\" (UniqueName: \"kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.863680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.863598    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:20.921081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:20.921027    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:21.282965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:21.277984    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"]
Nov 01 20:11:21.290792 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:11:21.286371    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6e32a5_a0a1_4c2f_9708_66361456e02d.slice/crio-9f7884dadb0ca99b7af748418596b91ad897d4379e5d60f8646b652a055eb156 WatchSource:0}: Error finding container 9f7884dadb0ca99b7af748418596b91ad897d4379e5d60f8646b652a055eb156: Status 404 returned error can't find the container with id 9f7884dadb0ca99b7af748418596b91ad897d4379e5d60f8646b652a055eb156
Nov 01 20:11:21.297072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:21.296635    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:11:21.339879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:21.339808    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:21.481705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:21.477482    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx" event={"ID":"7e6e32a5-a0a1-4c2f-9708-66361456e02d","Type":"ContainerStarted","Data":"9f7884dadb0ca99b7af748418596b91ad897d4379e5d60f8646b652a055eb156"}
Nov 01 20:11:21.532568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:21.532537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:22.040764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:22.040668    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:22.484550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:22.484503    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx" event={"ID":"7e6e32a5-a0a1-4c2f-9708-66361456e02d","Type":"ContainerStarted","Data":"bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe"}
Nov 01 20:11:22.484785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:22.484573    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:22.500554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:22.500466    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx" podStartSLOduration=2.053452 podStartE2EDuration="2.500447922s" totalImagesPullingTime="446.995922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:22.499504086 +0000 UTC m=+727.915136559" watchObservedRunningTime="2026-11-01 20:11:22.500447922 +0000 UTC m=+727.916080397"
Nov 01 20:11:22.644268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:22.644231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:23.322263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:23.322227    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:23.853546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:23.853505    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:24.130859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.130756    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm" podUID="d6b80844-b954-4f7b-b9d0-574164979e0a" containerName="payload-processing" containerID="cri-o://0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344" gracePeriod=25
Nov 01 20:11:24.435114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.435084    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:11:24.484294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.484250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:24.496997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.496964    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume\") pod \"d6b80844-b954-4f7b-b9d0-574164979e0a\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") "
Nov 01 20:11:24.497166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.497019    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5\") pod \"d6b80844-b954-4f7b-b9d0-574164979e0a\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") "
Nov 01 20:11:24.497166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.497065    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials\") pod \"d6b80844-b954-4f7b-b9d0-574164979e0a\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") "
Nov 01 20:11:24.497332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.497315    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume\") pod \"d6b80844-b954-4f7b-b9d0-574164979e0a\" (UID: \"d6b80844-b954-4f7b-b9d0-574164979e0a\") "
Nov 01 20:11:24.498252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.498226    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume" pod "d6b80844-b954-4f7b-b9d0-574164979e0a" (UID: "d6b80844-b954-4f7b-b9d0-574164979e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:24.498517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.498251    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume" pod "d6b80844-b954-4f7b-b9d0-574164979e0a" (UID: "d6b80844-b954-4f7b-b9d0-574164979e0a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:24.499731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.499690    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5" pod "d6b80844-b954-4f7b-b9d0-574164979e0a" (UID: "d6b80844-b954-4f7b-b9d0-574164979e0a"). InnerVolumeSpecName "kube-api-access-57vc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:24.501140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.501105    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials" pod "d6b80844-b954-4f7b-b9d0-574164979e0a" (UID: "d6b80844-b954-4f7b-b9d0-574164979e0a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:24.501140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.501108    6479 generic.go:356] "Generic (PLEG): container finished" podID="d6b80844-b954-4f7b-b9d0-574164979e0a" containerID="0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344" exitCode=0
Nov 01 20:11:24.501300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.501128    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm" event={"ID":"d6b80844-b954-4f7b-b9d0-574164979e0a","Type":"ContainerDied","Data":"0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344"}
Nov 01 20:11:24.501300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.501192    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"
Nov 01 20:11:24.501300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.501262    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm" event={"ID":"d6b80844-b954-4f7b-b9d0-574164979e0a","Type":"ContainerDied","Data":"f41951b0d9824aa86121138544adf0a094722a8c91660aa6bfdaf629a291058b"}
Nov 01 20:11:24.501451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.501305    6479 scope.go:122] "RemoveContainer" containerID="0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344"
Nov 01 20:11:24.524878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.524849    6479 scope.go:122] "RemoveContainer" containerID="0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344"
Nov 01 20:11:24.525221 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:24.525200    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344\": container with ID starting with 0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344 not found: ID does not exist" containerID="0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344"
Nov 01 20:11:24.525284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.525231    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344"} err="failed to get container status \"0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344\": rpc error: code = NotFound desc = could not find container \"0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344\": container with ID starting with 0f593aef0d4f0378aa677e7a15cc4b4649778c67f1af33feb96078b0728fc344 not found: ID does not exist"
Nov 01 20:11:24.535560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.535529    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"]
Nov 01 20:11:24.537864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.537839    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cp5mzm"]
Nov 01 20:11:24.605684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.605247    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:24.605684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.605292    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b80844-b954-4f7b-b9d0-574164979e0a-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:24.605684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.605308    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vc5\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-kube-api-access-57vc5\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:24.605684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.605323    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b80844-b954-4f7b-b9d0-574164979e0a-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:24.708328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:24.708294    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b80844-b954-4f7b-b9d0-574164979e0a" path="/var/lib/kubelet/pods/d6b80844-b954-4f7b-b9d0-574164979e0a/volumes"
Nov 01 20:11:25.051510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:25.051359    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:25.651092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:25.651059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:26.245328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:26.245280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:27.036290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:27.036241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:28.517876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:28.517837    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:11:29.224681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:29.208688    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:29.322708 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:29.321876    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:11:30.335339 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:30.335292    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:11:30.336432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:30.335986    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:30.366775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:30.366723    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:30.447309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:30.447230    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:30.707569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:30.707535    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:31.551436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:31.551364    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:32.262087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:32.261999    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:32.898356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:32.898316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:33.752779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:33.752743    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:34.424604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:34.424559    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:35.268672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:35.267575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:35.966993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:35.966957    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:36.680606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:36.680566    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:38.046609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:38.046571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:38.556724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:38.556685    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:39.140574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:39.140540    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:39.873669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:39.872790    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:40.186297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.186219    6479 generic.go:356] "Generic (PLEG): container finished" podID="5b3ab07f-1da6-442f-857b-5789828beea8" containerID="10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15" exitCode=0
Nov 01 20:11:40.186297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.186276    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" event={"ID":"5b3ab07f-1da6-442f-857b-5789828beea8","Type":"ContainerDied","Data":"10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"}
Nov 01 20:11:40.186729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.186687    6479 scope.go:122] "RemoveContainer" containerID="10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"
Nov 01 20:11:40.188275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.188242    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:40.188392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.188317    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:40.188392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.188349    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:11:40.446592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:40.446556    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:41.038608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.038574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:41.096023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.095978    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:41.199695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.199637    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" event={"ID":"5b3ab07f-1da6-442f-857b-5789828beea8","Type":"ContainerStarted","Data":"fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674"}
Nov 01 20:11:41.635985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.635955    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:41.773671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.762131    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:11:41.807665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.807384    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:41.807665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.807510    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:41.807665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:41.807551    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:42.250106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:42.250068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:42.367575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:42.367538    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:42.464248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:42.464194    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:42.577637 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:42.577537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:42.803211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:42.803171    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:42.803509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:42.803476    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:43.807169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:43.807127    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:43.807821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:43.807787    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:44.267897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:44.267846    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.376715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:44.376662    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.380977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:44.380930    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.447857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:44.447793    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.803108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:44.803071    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:44.803407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:44.803371    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:45.150724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.149795    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:45.279793 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.279752    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:11:45.317459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.317425    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:11:45.332859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.332792    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:11:45.379797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.379745    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:45.380848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.380823    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:45.402729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.382056    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:45.452324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.408274    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:11:45.492672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.452625    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:11:45.492672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.453094    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:45.820694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.812156    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:45.820694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:45.812216    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:46.322768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.322724    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:46.334029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.333990    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:11:46.334940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.334915    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:46.441826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.441777    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:46.764948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.459738    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:11:46.776674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.459777    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 01 20:11:46.870669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.484630    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:46.893943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.859775    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:46.901672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:46.894202    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:47.146313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.146222    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:47.247965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.247916    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:47.429723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.426698    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:11:47.430894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.427723    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:47.784668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.783025    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:47.788668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.785295    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 01 20:11:47.804552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.804431    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:47.804552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.804492    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:47.913672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:47.904844    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:48.031247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.031212    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:48.039858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.039785    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:11:48.202691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.198948    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:11:48.202691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.200043    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:48.202691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.201230    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:48.204335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.203294    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:48.204505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.204380    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:48.304882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.304795    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:48.834678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.834408    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:48.834678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.834483    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:48.853703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.849920    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"]
Nov 01 20:11:48.853703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.850757    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d6b80844-b954-4f7b-b9d0-574164979e0a" containerName="payload-processing"
Nov 01 20:11:48.853703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.850776    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b80844-b954-4f7b-b9d0-574164979e0a" containerName="payload-processing"
Nov 01 20:11:48.853703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.850926    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b80844-b954-4f7b-b9d0-574164979e0a" containerName="payload-processing"
Nov 01 20:11:48.897289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.881326    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"]
Nov 01 20:11:48.897289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.881454    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:48.905672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:48.899707    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-390f1256d6c4\""
Nov 01 20:11:49.002462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.002426    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.002642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.002566    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jtgf\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.002824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.002798    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.002886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.002838    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.104605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.104504    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.104605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.104556    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.104851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.104709    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.104851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.104813    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jtgf\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.105493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.105467    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.105925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.105902    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.107720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.107696    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.111120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.111100    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:49.112634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.112617    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jtgf\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.149991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.149965    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"]
Nov 01 20:11:49.194970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.193067    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.195200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.194963    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"]
Nov 01 20:11:49.197957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.197933    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e38ef4\""
Nov 01 20:11:49.198167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.198145    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"agc-cred-f2313541a738\""
Nov 01 20:11:49.198314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.198294    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"payload-processing-external-model-e2e--fb221618-dockercfg-dsjj2\""
Nov 01 20:11:49.198469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.198092    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"kube-root-ca.crt\""
Nov 01 20:11:49.198558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.197952    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"routing-overlay\""
Nov 01 20:11:49.198558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.198012    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e38ef4\"/\"openshift-service-ca.crt\""
Nov 01 20:11:49.211100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.211075    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:49.223751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.223730    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:49.312671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.311415    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.312671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.311640    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthv9\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.312671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.311778    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.312671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.311833    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.413884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.413793    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthv9\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.414200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.414180    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.414376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.414350    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.415186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.415142    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.416365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.416339    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.416498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.416347    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.449681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.431227    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.453639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.453541    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthv9\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9\") pod \"payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") " pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.599669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.577508    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:49.711786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.710468    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"]
Nov 01 20:11:49.802438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.802398    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:11:49.802586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.802456    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 20:11:49.915156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.915121    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:49.936082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:49.935864    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"]
Nov 01 20:11:50.198303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.198272    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"]
Nov 01 20:11:50.215056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.215018    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" event={"ID":"46b2df20-f2d8-48f1-b686-996b78787cf7","Type":"ContainerStarted","Data":"707c6bc2629f38eb28120942433ed97ef98774776ac621842f31182d614ff5e3"}
Nov 01 20:11:50.215234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.215079    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" event={"ID":"46b2df20-f2d8-48f1-b686-996b78787cf7","Type":"ContainerStarted","Data":"2619dc64608a022e988b4719c0ce48474b3fb975ceffd41ec89aad69ae645d85"}
Nov 01 20:11:50.215234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.215138    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:50.215358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.215281    6479 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" secret="" err="secret \"payload-processing-external-model-e2e--fb221618-dockercfg-dsjj2\" not found"
Nov 01 20:11:50.217254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.217225    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" event={"ID":"012eb06f-31f7-4927-992a-fe38f768b156","Type":"ContainerStarted","Data":"083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655"}
Nov 01 20:11:50.217394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.217286    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:50.217394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.217312    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" event={"ID":"012eb06f-31f7-4927-992a-fe38f768b156","Type":"ContainerStarted","Data":"6624f7795a973877f33aaf4c6ca992b269cbcec1173e62156f4f164d2e6c19fd"}
Nov 01 20:11:50.219297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.219226    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:50.235472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.235422    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" podStartSLOduration=1.235406703 podStartE2EDuration="1.235406703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:50.232477664 +0000 UTC m=+755.648110140" watchObservedRunningTime="2026-11-01 20:11:50.235406703 +0000 UTC m=+755.651039176"
Nov 01 20:11:50.250437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.250390    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" podStartSLOduration=2.250378054 podStartE2EDuration="2.250378054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:50.24943946 +0000 UTC m=+755.665071934" watchObservedRunningTime="2026-11-01 20:11:50.250378054 +0000 UTC m=+755.666010526"
Nov 01 20:11:50.318961 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.318932    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:50.319144 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319000    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:50.818980003 +0000 UTC m=+756.234612475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:50.319144 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319014    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-f2313541a738: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:50.319144 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319034    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:50.319144 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319089    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:50.819070876 +0000 UTC m=+756.234703334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : secret "agc-cred-f2313541a738" not found
Nov 01 20:11:50.319402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.319232    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:50.319402 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319374    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5370bb/agc-cred-390f1256d6c4: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:50.319402 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319396    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:50.319520 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.319443    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials podName:012eb06f-31f7-4927-992a-fe38f768b156 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:50.819431103 +0000 UTC m=+756.235063556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" (UID: "012eb06f-31f7-4927-992a-fe38f768b156") : secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:50.609726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.607581    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"]
Nov 01 20:11:50.811930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.811901    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:50.813892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.813870    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:50.816890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:50.816869    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.842697    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.842785    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:51.842765666 +0000 UTC m=+757.258398199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.843628    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5370bb/agc-cred-390f1256d6c4: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.843670    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.843716    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials podName:012eb06f-31f7-4927-992a-fe38f768b156 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:51.843702508 +0000 UTC m=+757.259334963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" (UID: "012eb06f-31f7-4927-992a-fe38f768b156") : secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.843788    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-f2313541a738: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.843798    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:50.847895 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:50.843829    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:51.843818942 +0000 UTC m=+757.259451407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : secret "agc-cred-f2313541a738" not found
Nov 01 20:11:51.149100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:51.149069    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:51.871771 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.871738    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:51.872055 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.872037    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:53.87201677 +0000 UTC m=+759.287649225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:51.872540 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.871908    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5370bb/agc-cred-390f1256d6c4: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:51.872684 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.872669    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:51.872833 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.872821    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials podName:012eb06f-31f7-4927-992a-fe38f768b156 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:53.87280707 +0000 UTC m=+759.288439522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" (UID: "012eb06f-31f7-4927-992a-fe38f768b156") : secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:51.872940 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.871959    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-f2313541a738: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:51.873042 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.873030    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:51.873173 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:51.873162    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:53.873147027 +0000 UTC m=+759.288779480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : secret "agc-cred-f2313541a738" not found
Nov 01 20:11:52.380543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:52.380504    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:52.699311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:52.699280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:53.149732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:53.134361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:53.436333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:53.435890    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:11:53.803812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:53.803747    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:11:53.933205 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.933164    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e38ef4/agc-cred-f2313541a738: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:53.933439 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.933423    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m: secret "agc-cred-f2313541a738" not found
Nov 01 20:11:53.933613 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.933599    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:57.933577421 +0000 UTC m=+763.349209889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : secret "agc-cred-f2313541a738" not found
Nov 01 20:11:53.934199 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.934176    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5370bb/agc-cred-390f1256d6c4: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:53.934339 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.934326    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp: secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:53.934470 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.934458    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials podName:012eb06f-31f7-4927-992a-fe38f768b156 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:57.934441853 +0000 UTC m=+763.350074321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" (UID: "012eb06f-31f7-4927-992a-fe38f768b156") : secret "agc-cred-390f1256d6c4" not found
Nov 01 20:11:53.934608 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.934595    6479 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-plugins-e2e-extmod-a-e38ef4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:53.934757 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:53.934744    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume podName:46b2df20-f2d8-48f1-b686-996b78787cf7 nodeName:}" failed. No retries permitted until 2026-11-01 20:11:57.934729325 +0000 UTC m=+763.350361794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e38ef4" not found
Nov 01 20:11:54.060914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:54.060811    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:54.484672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:54.483794    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:54.484672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:54.484440    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:54.892929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:54.892825    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:54.905677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:54.899874    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:55.114668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.090788    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:11:55.130832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.109113    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:55.144751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.127795    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:55.332309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.332245    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:55.434167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.434073    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:11:55.488764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.488621    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:55.601219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.601180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:55.666707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:55.666149    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:11:56.151410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:56.151375    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:11:56.288669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:56.285723    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" podUID="46b2df20-f2d8-48f1-b686-996b78787cf7" containerName="payload-processing" containerID="cri-o://707c6bc2629f38eb28120942433ed97ef98774776ac621842f31182d614ff5e3" gracePeriod=25
Nov 01 20:11:56.971133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:56.970378    6479 generic.go:356] "Generic (PLEG): container finished" podID="46b2df20-f2d8-48f1-b686-996b78787cf7" containerID="707c6bc2629f38eb28120942433ed97ef98774776ac621842f31182d614ff5e3" exitCode=0
Nov 01 20:11:56.971133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:56.970582    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" event={"ID":"46b2df20-f2d8-48f1-b686-996b78787cf7","Type":"ContainerDied","Data":"707c6bc2629f38eb28120942433ed97ef98774776ac621842f31182d614ff5e3"}
Nov 01 20:11:57.243558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.243474    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:57.308181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.296747    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" podUID="012eb06f-31f7-4927-992a-fe38f768b156" containerName="payload-processing" containerID="cri-o://083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655" gracePeriod=25
Nov 01 20:11:57.365691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.360166    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume\") pod \"46b2df20-f2d8-48f1-b686-996b78787cf7\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") "
Nov 01 20:11:57.365691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.360256    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9\") pod \"46b2df20-f2d8-48f1-b686-996b78787cf7\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") "
Nov 01 20:11:57.365691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.360392    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume\") pod \"46b2df20-f2d8-48f1-b686-996b78787cf7\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") "
Nov 01 20:11:57.365691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.360521    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials\") pod \"46b2df20-f2d8-48f1-b686-996b78787cf7\" (UID: \"46b2df20-f2d8-48f1-b686-996b78787cf7\") "
Nov 01 20:11:57.365691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.363712    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume" pod "46b2df20-f2d8-48f1-b686-996b78787cf7" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:57.374268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.373721    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume" pod "46b2df20-f2d8-48f1-b686-996b78787cf7" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:57.377018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.376965    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials" pod "46b2df20-f2d8-48f1-b686-996b78787cf7" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:57.377355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.377330    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9" pod "46b2df20-f2d8-48f1-b686-996b78787cf7" (UID: "46b2df20-f2d8-48f1-b686-996b78787cf7"). InnerVolumeSpecName "kube-api-access-zthv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:57.460472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.460439    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:57.462135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.462101    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.462135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.462135    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.462335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.462153    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthv9\" (UniqueName: \"kubernetes.io/projected/46b2df20-f2d8-48f1-b686-996b78787cf7-kube-api-access-zthv9\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.462335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.462167    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b2df20-f2d8-48f1-b686-996b78787cf7-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.706736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.706698    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:57.869069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.868977    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume\") pod \"012eb06f-31f7-4927-992a-fe38f768b156\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") "
Nov 01 20:11:57.869232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.869203    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf\") pod \"012eb06f-31f7-4927-992a-fe38f768b156\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") "
Nov 01 20:11:57.869285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.869255    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume\") pod \"012eb06f-31f7-4927-992a-fe38f768b156\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") "
Nov 01 20:11:57.869396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.869365    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume" pod "012eb06f-31f7-4927-992a-fe38f768b156" (UID: "012eb06f-31f7-4927-992a-fe38f768b156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:57.869396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.869385    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials\") pod \"012eb06f-31f7-4927-992a-fe38f768b156\" (UID: \"012eb06f-31f7-4927-992a-fe38f768b156\") "
Nov 01 20:11:57.869678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.869625    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume" pod "012eb06f-31f7-4927-992a-fe38f768b156" (UID: "012eb06f-31f7-4927-992a-fe38f768b156"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:57.870032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.870009    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.870148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.870034    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/012eb06f-31f7-4927-992a-fe38f768b156-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.871749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.871455    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials" pod "012eb06f-31f7-4927-992a-fe38f768b156" (UID: "012eb06f-31f7-4927-992a-fe38f768b156"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:57.871749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.871455    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf" pod "012eb06f-31f7-4927-992a-fe38f768b156" (UID: "012eb06f-31f7-4927-992a-fe38f768b156"). InnerVolumeSpecName "kube-api-access-2jtgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:57.970843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.970801    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jtgf\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-kube-api-access-2jtgf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.970843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.970833    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/012eb06f-31f7-4927-992a-fe38f768b156-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:57.977420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.977389    6479 generic.go:356] "Generic (PLEG): container finished" podID="012eb06f-31f7-4927-992a-fe38f768b156" containerID="083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655" exitCode=0
Nov 01 20:11:57.977909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.977461    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"
Nov 01 20:11:57.977909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.977476    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" event={"ID":"012eb06f-31f7-4927-992a-fe38f768b156","Type":"ContainerDied","Data":"083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655"}
Nov 01 20:11:57.977909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.977535    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp" event={"ID":"012eb06f-31f7-4927-992a-fe38f768b156","Type":"ContainerDied","Data":"6624f7795a973877f33aaf4c6ca992b269cbcec1173e62156f4f164d2e6c19fd"}
Nov 01 20:11:57.977909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.977557    6479 scope.go:122] "RemoveContainer" containerID="083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655"
Nov 01 20:11:57.979170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.979144    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m" event={"ID":"46b2df20-f2d8-48f1-b686-996b78787cf7","Type":"ContainerDied","Data":"2619dc64608a022e988b4719c0ce48474b3fb975ceffd41ec89aad69ae645d85"}
Nov 01 20:11:57.979170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.979169    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"
Nov 01 20:11:57.989210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.989120    6479 scope.go:122] "RemoveContainer" containerID="083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655"
Nov 01 20:11:57.989827 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:57.989687    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655\": container with ID starting with 083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655 not found: ID does not exist" containerID="083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655"
Nov 01 20:11:57.989827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.989724    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655"} err="failed to get container status \"083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655\": rpc error: code = NotFound desc = could not find container \"083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655\": container with ID starting with 083d3f9749351c0b3d30da152fbc346d2872437a624f41aad52f8e6f6ae41655 not found: ID does not exist"
Nov 01 20:11:57.989827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:57.989748    6479 scope.go:122] "RemoveContainer" containerID="707c6bc2629f38eb28120942433ed97ef98774776ac621842f31182d614ff5e3"
Nov 01 20:11:58.001265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.001244    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"]
Nov 01 20:11:58.010699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.010674    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-74d7plvjp"]
Nov 01 20:11:58.017906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.017885    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"]
Nov 01 20:11:58.022212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.022188    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e38ef4/payload-processing-external-model-e2e-extmod-a-e38ef4-bfd4pw85m"]
Nov 01 20:11:58.545837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.545801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:11:58.710787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.710752    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012eb06f-31f7-4927-992a-fe38f768b156" path="/var/lib/kubelet/pods/012eb06f-31f7-4927-992a-fe38f768b156/volumes"
Nov 01 20:11:58.711167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:58.711154    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b2df20-f2d8-48f1-b686-996b78787cf7" path="/var/lib/kubelet/pods/46b2df20-f2d8-48f1-b686-996b78787cf7/volumes"
Nov 01 20:11:59.172569 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:11:59.172527    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:11:59.360375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:11:59.360326    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:00.121825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:00.121795    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:00.877693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:00.877657    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:01.569883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:01.569844    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:02.438803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:02.438769    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:03.140993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:03.140956    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:03.672708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:03.672672    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:12:03.769926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:03.769889    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:04.610967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:04.610905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:04.769673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:04.767247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:04.863591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:04.863497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:04.978827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:04.978794    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:05.647172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:05.647133    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:05.762676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:05.758277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:06.327813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.327762    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:06.415197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.415155    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:06.456208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.456177    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:06.485058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.484991    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:06.574911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.574875    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:06.690548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.690426    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:12:06.765798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:06.765750    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:07.182671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.179136    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:07.498335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.498291    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:07.629638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.629603    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:12:07.703309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.703266    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:07.716699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.716101    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:12:07.872897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.872799    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"]
Nov 01 20:12:07.979682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:07.974355    6479 cpu_manager.go:414] "containerMap: removing container" podUID="46b2df20-f2d8-48f1-b686-996b78787cf7" containerName="payload-processing"
Nov 01 20:12:08.000074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.000040    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b2df20-f2d8-48f1-b686-996b78787cf7" containerName="payload-processing"
Nov 01 20:12:08.000324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.000306    6479 cpu_manager.go:414] "containerMap: removing container" podUID="012eb06f-31f7-4927-992a-fe38f768b156" containerName="payload-processing"
Nov 01 20:12:08.048787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.048754    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012eb06f-31f7-4927-992a-fe38f768b156" containerName="payload-processing"
Nov 01 20:12:08.049204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.049179    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="012eb06f-31f7-4927-992a-fe38f768b156" containerName="payload-processing"
Nov 01 20:12:08.192894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.192807    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b2df20-f2d8-48f1-b686-996b78787cf7" containerName="payload-processing"
Nov 01 20:12:08.241667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.219262    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:12:08.264666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.261832    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"]
Nov 01 20:12:08.264666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.262020    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.311857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.275485    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:12:08.311857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.275837    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-92ebf408b9f6\""
Nov 01 20:12:08.311857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.291143    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:12:08.313534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.075734    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:08.331592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.325794    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:12:08.440918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.440882    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:08.446304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.446268    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.446525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.446504    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.446830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.446806    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.447017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.446997    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2894\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.459109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.459086    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:12:08.551025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.550436    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.551025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.550516    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2894\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.551025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.550719    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.551025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.550784    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.551909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.551870    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.552639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.552616    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.574604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.574560    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.580070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.579605    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2894\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.745295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.745196    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:08.816927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:08.816767    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:09.025889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:09.025858    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"]
Nov 01 20:12:09.027619 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:12:09.027592    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39f2d03_9333_41f8_b201_ed89f200dfc5.slice/crio-a39b76930a84d804d73e888f9f64b30e6984ed38f35370d768620110c10bb082 WatchSource:0}: Error finding container a39b76930a84d804d73e888f9f64b30e6984ed38f35370d768620110c10bb082: Status 404 returned error can't find the container with id a39b76930a84d804d73e888f9f64b30e6984ed38f35370d768620110c10bb082
Nov 01 20:12:09.399970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:09.399873    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf" event={"ID":"e39f2d03-9333-41f8-b201-ed89f200dfc5","Type":"ContainerStarted","Data":"b842307d502e7b501aec23142b4553eee8b851d53dab4fce4fcf435e75341528"}
Nov 01 20:12:09.399970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:09.399940    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf" event={"ID":"e39f2d03-9333-41f8-b201-ed89f200dfc5","Type":"ContainerStarted","Data":"a39b76930a84d804d73e888f9f64b30e6984ed38f35370d768620110c10bb082"}
Nov 01 20:12:09.399970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:09.399962    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:09.416787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:09.416731    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf" podStartSLOduration=2.4167167259999998 podStartE2EDuration="2.416716726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:09.414161119 +0000 UTC m=+774.829793592" watchObservedRunningTime="2026-11-01 20:12:09.416716726 +0000 UTC m=+774.832349198"
Nov 01 20:12:09.478208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:09.478156    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:10.353866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:10.353832    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:10.974374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:10.974334    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:11.695298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:11.695266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:12.264845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:12.264799    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:12.941426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:12.941394    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:13.544536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:13.544504    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:14.151697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:14.151628    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:14.740450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:14.740409    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:15.335408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:15.335372    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:15.955027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:15.954996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:16.537892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:16.537863    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:17.135803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:17.135756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:17.762286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:17.762250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:18.438230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:18.438201    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:19.070667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:19.070605    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:19.657591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:19.657551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:20.411685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:20.411632    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:20.489053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:20.489018    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"]
Nov 01 20:12:20.502571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:20.502538    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:21.296143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:21.296105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:21.666689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:21.666583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:22.440429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:22.440382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:23.577961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:23.577922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:24.461468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:24.461433    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:25.256359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.256321    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:25.490143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.490107    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4" podUID="b4c3520b-1ca1-4844-a37b-dbcdbd062872" containerName="payload-processing" containerID="cri-o://ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff" gracePeriod=25
Nov 01 20:12:25.768057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.767996    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:12:25.936567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.936534    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials\") pod \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") "
Nov 01 20:12:25.936802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.936597    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume\") pod \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") "
Nov 01 20:12:25.936877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.936859    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7\") pod \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") "
Nov 01 20:12:25.937054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.936942    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume\") pod \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\" (UID: \"b4c3520b-1ca1-4844-a37b-dbcdbd062872\") "
Nov 01 20:12:25.937054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.937011    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume" pod "b4c3520b-1ca1-4844-a37b-dbcdbd062872" (UID: "b4c3520b-1ca1-4844-a37b-dbcdbd062872"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:25.937463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.937437    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:25.937590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.937441    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume" pod "b4c3520b-1ca1-4844-a37b-dbcdbd062872" (UID: "b4c3520b-1ca1-4844-a37b-dbcdbd062872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:25.939332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.939299    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7" pod "b4c3520b-1ca1-4844-a37b-dbcdbd062872" (UID: "b4c3520b-1ca1-4844-a37b-dbcdbd062872"). InnerVolumeSpecName "kube-api-access-7g4p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:25.939332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.939321    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials" pod "b4c3520b-1ca1-4844-a37b-dbcdbd062872" (UID: "b4c3520b-1ca1-4844-a37b-dbcdbd062872"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:25.951831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:25.951806    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:26.042723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.042031    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4p7\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-kube-api-access-7g4p7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:26.042723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.042076    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c3520b-1ca1-4844-a37b-dbcdbd062872-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:26.042723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.042093    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4c3520b-1ca1-4844-a37b-dbcdbd062872-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:26.597290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.597250    6479 generic.go:356] "Generic (PLEG): container finished" podID="b4c3520b-1ca1-4844-a37b-dbcdbd062872" containerID="ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff" exitCode=0
Nov 01 20:12:26.597805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.597301    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"
Nov 01 20:12:26.597805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.597335    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4" event={"ID":"b4c3520b-1ca1-4844-a37b-dbcdbd062872","Type":"ContainerDied","Data":"ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff"}
Nov 01 20:12:26.597805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.597408    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4" event={"ID":"b4c3520b-1ca1-4844-a37b-dbcdbd062872","Type":"ContainerDied","Data":"1e05b488e32ae1805988d5888e3692b8c3a3e6432a690129a5c4992170c15360"}
Nov 01 20:12:26.597805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.597441    6479 scope.go:122] "RemoveContainer" containerID="ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff"
Nov 01 20:12:26.607775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.607754    6479 scope.go:122] "RemoveContainer" containerID="ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff"
Nov 01 20:12:26.608036 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:12:26.608017    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff\": container with ID starting with ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff not found: ID does not exist" containerID="ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff"
Nov 01 20:12:26.608106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.608046    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff"} err="failed to get container status \"ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff\": rpc error: code = NotFound desc = could not find container \"ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff\": container with ID starting with ce062063f5b9da3e5874ce53865dd8002845751742e5f5fc22b57db52b3852ff not found: ID does not exist"
Nov 01 20:12:26.621159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.621126    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"]
Nov 01 20:12:26.627902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.627878    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5pnzg4"]
Nov 01 20:12:26.669194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.669159    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:26.708545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:26.708517    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c3520b-1ca1-4844-a37b-dbcdbd062872" path="/var/lib/kubelet/pods/b4c3520b-1ca1-4844-a37b-dbcdbd062872/volumes"
Nov 01 20:12:27.083482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:27.083448    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:27.352281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:27.352198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:28.054806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:28.054769    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:28.854438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:28.854394    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:29.173684 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:12:29.173560    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:12:29.533732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:29.533696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:29.638194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:29.638150    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:30.178227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.178187    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"]
Nov 01 20:12:30.179074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.179052    6479 cpu_manager.go:414] "containerMap: removing container" podUID="b4c3520b-1ca1-4844-a37b-dbcdbd062872" containerName="payload-processing"
Nov 01 20:12:30.179125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.179077    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c3520b-1ca1-4844-a37b-dbcdbd062872" containerName="payload-processing"
Nov 01 20:12:30.179249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.179236    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c3520b-1ca1-4844-a37b-dbcdbd062872" containerName="payload-processing"
Nov 01 20:12:30.183213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.183191    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.188762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.188731    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"]
Nov 01 20:12:30.261908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.261865    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-8a2091566ddc\""
Nov 01 20:12:30.307216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.307177    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5l24\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.307377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.307236    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.307447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.307398    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.307519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.307500    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.409109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.409059    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5l24\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.409300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.409135    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.409300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.409290    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.409415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.409363    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.410086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.410055    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.410210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.410150    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.412143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.412122    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.417806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.417779    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5l24\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.463755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.463704    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:30.508590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.508262    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:30.837787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:30.837753    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"]
Nov 01 20:12:30.839093 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:12:30.839067    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4366a078_9273_4995_a29c_d2021ea006b5.slice/crio-1d6ecb7f6c580b4ba95852240a1eee5cb752435df3c7e78c72dbec36f66f6ac9 WatchSource:0}: Error finding container 1d6ecb7f6c580b4ba95852240a1eee5cb752435df3c7e78c72dbec36f66f6ac9: Status 404 returned error can't find the container with id 1d6ecb7f6c580b4ba95852240a1eee5cb752435df3c7e78c72dbec36f66f6ac9
Nov 01 20:12:31.142685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.142568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:31.346269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.346234    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:12:31.630053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.630016    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" event={"ID":"4366a078-9273-4995-a29c-d2021ea006b5","Type":"ContainerStarted","Data":"25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3"}
Nov 01 20:12:31.630251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.630077    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" event={"ID":"4366a078-9273-4995-a29c-d2021ea006b5","Type":"ContainerStarted","Data":"1d6ecb7f6c580b4ba95852240a1eee5cb752435df3c7e78c72dbec36f66f6ac9"}
Nov 01 20:12:31.630251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.630105    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:31.650963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.650912    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" podStartSLOduration=1.6509000459999998 podStartE2EDuration="1.650900046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:31.647910041 +0000 UTC m=+797.063542514" watchObservedRunningTime="2026-11-01 20:12:31.650900046 +0000 UTC m=+797.066532518"
Nov 01 20:12:31.905668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:31.905570    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:33.055673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:33.054591    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:34.158670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:34.156770    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:34.269669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:34.268792    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:34.319671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:34.318449    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:34.343671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:34.326204    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:34.648670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:34.639144    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:35.603361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:35.603320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:36.261610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:36.261572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:36.855961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:36.855929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:37.457368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:37.457333    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:38.206604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:38.206573    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:38.817890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:38.817855    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:39.431080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:39.431044    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:40.149475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:40.149437    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:40.749310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:40.749260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:41.454821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:41.454788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:42.056529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:42.056494    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:42.665876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:42.665816    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:42.684861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:42.684828    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:12:42.768474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:42.768438    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"]
Nov 01 20:12:43.283007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:43.282965    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:44.047290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:44.047248    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:44.659260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:44.659224    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:45.440890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:45.440858    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:46.438895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:46.438858    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:47.420022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:47.419971    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:47.794249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:47.794211    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"]
Nov 01 20:12:47.888671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:47.875507    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf" podUID="e39f2d03-9333-41f8-b201-ed89f200dfc5" containerName="payload-processing" containerID="cri-o://b842307d502e7b501aec23142b4553eee8b851d53dab4fce4fcf435e75341528" gracePeriod=25
Nov 01 20:12:47.983779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:47.959793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:48.045139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.004317    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"]
Nov 01 20:12:48.045139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.004668    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.048580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.048540    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.048869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.048841    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.049104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.049083    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.053895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.049276    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.054144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.054117    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9sv\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.057757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.057730    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.057957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.057938    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.061746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.061720    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.061924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.061904    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.086404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.086367    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-29b57c-openshift-default-dockercfg-6fnqw\""
Nov 01 20:12:48.161536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.160784    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180629    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9sv\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180703    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180749    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180783    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180815    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180883    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.180915    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.181024    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.181093    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.182078    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.184099    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.189834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.184422    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.190773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.189990    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.210915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.206928    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.299727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.274630    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.300020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.277285    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:48.300705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.300667    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:48.323399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.323356    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9sv\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.337788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.337728    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:12:48.387678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.384185    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.387678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.384712    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:48.459855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.459807    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:48.540994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:48.540957    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:49.039669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.035309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:49.188780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.188734    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 20:12:49.203101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.199311    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:12:49.331084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.330992    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 01 20:12:49.331860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.331834    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:12:49.352061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.332059    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:49.379972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.352320    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:49.453854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.380234    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:12:49.454152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.454133    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:12:49.464672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.461032    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:49.500144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.499886    6479 generic.go:356] "Generic (PLEG): container finished" podID="e39f2d03-9333-41f8-b201-ed89f200dfc5" containerID="b842307d502e7b501aec23142b4553eee8b851d53dab4fce4fcf435e75341528" exitCode=0
Nov 01 20:12:49.500144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.500074    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf" event={"ID":"e39f2d03-9333-41f8-b201-ed89f200dfc5","Type":"ContainerDied","Data":"b842307d502e7b501aec23142b4553eee8b851d53dab4fce4fcf435e75341528"}
Nov 01 20:12:49.511387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.510797    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:12:49.516467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:49.516436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:50.025158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.019615    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:12:50.116028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.115997    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:50.171615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.171581    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:12:50.199668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.198026    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:50.218891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.218195    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:12:50.228668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.219108    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:12:50.253872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.253847    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:12:50.277134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.276412    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume\") pod \"e39f2d03-9333-41f8-b201-ed89f200dfc5\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") "
Nov 01 20:12:50.277134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.276516    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials\") pod \"e39f2d03-9333-41f8-b201-ed89f200dfc5\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") "
Nov 01 20:12:50.277134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.276565    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894\") pod \"e39f2d03-9333-41f8-b201-ed89f200dfc5\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") "
Nov 01 20:12:50.277134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.276632    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume\") pod \"e39f2d03-9333-41f8-b201-ed89f200dfc5\" (UID: \"e39f2d03-9333-41f8-b201-ed89f200dfc5\") "
Nov 01 20:12:50.277519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.277498    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume" pod "e39f2d03-9333-41f8-b201-ed89f200dfc5" (UID: "e39f2d03-9333-41f8-b201-ed89f200dfc5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:50.283677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.280734    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume" pod "e39f2d03-9333-41f8-b201-ed89f200dfc5" (UID: "e39f2d03-9333-41f8-b201-ed89f200dfc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:50.283677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.282290    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894" pod "e39f2d03-9333-41f8-b201-ed89f200dfc5" (UID: "e39f2d03-9333-41f8-b201-ed89f200dfc5"). InnerVolumeSpecName "kube-api-access-x2894". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:50.283973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.283952    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials" pod "e39f2d03-9333-41f8-b201-ed89f200dfc5" (UID: "e39f2d03-9333-41f8-b201-ed89f200dfc5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:50.338029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.337995    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"]
Nov 01 20:12:50.340164 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:12:50.340132    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af32888_a7a6_444e_be15_22035c3373ac.slice/crio-b16fc3371159af41faa1f4fe5026c446297c896d51de108caf8002bb1abfc106 WatchSource:0}: Error finding container b16fc3371159af41faa1f4fe5026c446297c896d51de108caf8002bb1abfc106: Status 404 returned error can't find the container with id b16fc3371159af41faa1f4fe5026c446297c896d51de108caf8002bb1abfc106
Nov 01 20:12:50.342923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.342895    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:12:50.343000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.342976    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:12:50.343044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.343005    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:12:50.378058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.378026    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:50.378058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.378053    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:50.378214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.378071    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2894\" (UniqueName: \"kubernetes.io/projected/e39f2d03-9333-41f8-b201-ed89f200dfc5-kube-api-access-x2894\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:50.378214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.378084    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39f2d03-9333-41f8-b201-ed89f200dfc5-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:50.507570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.507534    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf" event={"ID":"e39f2d03-9333-41f8-b201-ed89f200dfc5","Type":"ContainerDied","Data":"a39b76930a84d804d73e888f9f64b30e6984ed38f35370d768620110c10bb082"}
Nov 01 20:12:50.507753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.507595    6479 scope.go:122] "RemoveContainer" containerID="b842307d502e7b501aec23142b4553eee8b851d53dab4fce4fcf435e75341528"
Nov 01 20:12:50.507753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.507745    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"
Nov 01 20:12:50.511510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.511475    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" event={"ID":"9af32888-a7a6-444e-be15-22035c3373ac","Type":"ContainerStarted","Data":"ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"}
Nov 01 20:12:50.511616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.511540    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" event={"ID":"9af32888-a7a6-444e-be15-22035c3373ac","Type":"ContainerStarted","Data":"b16fc3371159af41faa1f4fe5026c446297c896d51de108caf8002bb1abfc106"}
Nov 01 20:12:50.539706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.539600    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podStartSLOduration=3.5395817149999997 podStartE2EDuration="3.539581715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:50.535104356 +0000 UTC m=+815.950736828" watchObservedRunningTime="2026-11-01 20:12:50.539581715 +0000 UTC m=+815.955214188"
Nov 01 20:12:50.546724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.546700    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:50.561742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.559618    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"]
Nov 01 20:12:50.561742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.559689    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxszjf"]
Nov 01 20:12:50.743670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.735499    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39f2d03-9333-41f8-b201-ed89f200dfc5" path="/var/lib/kubelet/pods/e39f2d03-9333-41f8-b201-ed89f200dfc5/volumes"
Nov 01 20:12:50.865139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:50.865047    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:51.551683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:51.551621    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:52.123542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:52.123508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:52.524924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:52.524891    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:52.525766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:52.525745    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:12:53.039859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:53.039824    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:53.135627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:53.135546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:53.678082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:53.678045    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:54.419165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.419132    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:54.541357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.539455    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:54.644042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.644008    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:54.798178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.798139    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"]
Nov 01 20:12:54.798807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.798778    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e39f2d03-9333-41f8-b201-ed89f200dfc5" containerName="payload-processing"
Nov 01 20:12:54.798807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.798798    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39f2d03-9333-41f8-b201-ed89f200dfc5" containerName="payload-processing"
Nov 01 20:12:54.799021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.798920    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39f2d03-9333-41f8-b201-ed89f200dfc5" containerName="payload-processing"
Nov 01 20:12:54.804185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.804163    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:54.818529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.818502    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"]
Nov 01 20:12:54.884214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.880800    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-29b57c-dockercfg-7dpnq\""
Nov 01 20:12:54.949669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.937165    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-29b57c\""
Nov 01 20:12:54.951404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.951371    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:54.954236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.952467    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwrk\" (UniqueName: \"kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk\") pod \"payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:54.954608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:54.954586    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.057462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.057377    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwrk\" (UniqueName: \"kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk\") pod \"payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.057837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.057816    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.059081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.059060    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.083728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.082450    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwrk\" (UniqueName: \"kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk\") pod \"payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.120337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.120303    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.408387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.408314    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"]
Nov 01 20:12:55.414393 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:12:55.413435    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78512e90_9f27_4cee_8655_c4bcb41ea1ea.slice/crio-616c922b0c5283481993f6ac85d8ae4de78ca1b0c70a09ecfd7ae5c6807061f3 WatchSource:0}: Error finding container 616c922b0c5283481993f6ac85d8ae4de78ca1b0c70a09ecfd7ae5c6807061f3: Status 404 returned error can't find the container with id 616c922b0c5283481993f6ac85d8ae4de78ca1b0c70a09ecfd7ae5c6807061f3
Nov 01 20:12:55.457346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.457316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:55.576669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.569969    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8" event={"ID":"78512e90-9f27-4cee-8655-c4bcb41ea1ea","Type":"ContainerStarted","Data":"44bebd3002e5150c2ea4dd8b20880a2a88eaf330c154c08fd42ce796c31385e5"}
Nov 01 20:12:55.576669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.570041    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8" event={"ID":"78512e90-9f27-4cee-8655-c4bcb41ea1ea","Type":"ContainerStarted","Data":"616c922b0c5283481993f6ac85d8ae4de78ca1b0c70a09ecfd7ae5c6807061f3"}
Nov 01 20:12:55.576669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.571040    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:12:55.589676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:55.589264    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8" podStartSLOduration=1.589243264 podStartE2EDuration="1.589243264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:55.586462994 +0000 UTC m=+821.002095480" watchObservedRunningTime="2026-11-01 20:12:55.589243264 +0000 UTC m=+821.004875736"
Nov 01 20:12:56.128556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:56.128520    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:56.765569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:56.765533    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:57.454426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:57.454385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:58.058933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:58.058898    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:58.751566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:58.751530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:58.851502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:58.851468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:12:59.170898 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:12:59.170809    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:12:59.631668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:12:59.629480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:00.178515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:00.178484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:00.279570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:00.279526    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:00.717181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:00.717148    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:00.947756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:00.947698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:01.452288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:01.452252    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:02.048853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:02.048809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:02.511631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:02.511595    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:13:02.661081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:02.661046    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:03.412433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:03.412398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:03.775334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:03.775242    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:03.882630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:03.882592    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:04.224182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:04.224145    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:05.038694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.036745    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 01 20:13:05.096683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.095150    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:05.125670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.124854    6479 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded"
Nov 01 20:13:05.182283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.182230    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:05.210241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.210198    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:05.227871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.227825    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:05.264815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.264762    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:05.398855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.398754    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:13:05.399114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.399084    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:05.512834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.512796    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:13:05.513126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.513097    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:05.880683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:05.867905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:06.455034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:06.454998    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:06.649878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:06.649845    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:06.760079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:06.759979    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:06.848567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:06.848532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:07.251228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:07.251199    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:07.603549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:07.603434    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:07.606323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:07.606299    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:13:07.631779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:07.631742    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"]
Nov 01 20:13:07.705635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:07.705600    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:07.972949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:07.972913    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:08.645244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:08.645215    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:08.875139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:08.875090    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:09.298265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:09.298230    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:09.560271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:09.560183    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:09.796520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:09.796468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:10.204238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.204203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:10.281223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.281188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:10.437313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.437280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:10.537884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.537749    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:10.856783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.856699    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"]
Nov 01 20:13:10.861339 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.861309    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.877035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.877011    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ebb0cff-openshift-default-dockercfg-74ktq\""
Nov 01 20:13:10.882395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.882331    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"]
Nov 01 20:13:10.909101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.909053    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:10.941709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.941676    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.941885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.941734    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942201    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942353    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942396    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942443    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5g8\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942483    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942550    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:10.942606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:10.942582    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.044876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.044834    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.064585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.064522    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.064948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.064925    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.065571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.065186    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5g8\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.065955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.065769    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.066357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.066336    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.067501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.067475    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.078745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.066853    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.079762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.077955    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.080070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.079394    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.080674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.080227    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.081057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.081036    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.086929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.079696    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.086929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.080582    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.105704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.103243    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.105704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.105293    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.126909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.124470    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5g8\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.143670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.136974    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy\") pod \"e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") " pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.193687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.191884    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:11.216189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.214721    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:11.303035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.302995    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:11.526565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.526536    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"]
Nov 01 20:13:11.528007 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:11.527981    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ad8539_02cd_4b6e_8d7a_d8bf72644ab4.slice/crio-0c9eeb4dc0967c4c67351600f73d91d9547607debd36b3e9a954c4a6be46646b WatchSource:0}: Error finding container 0c9eeb4dc0967c4c67351600f73d91d9547607debd36b3e9a954c4a6be46646b: Status 404 returned error can't find the container with id 0c9eeb4dc0967c4c67351600f73d91d9547607debd36b3e9a954c4a6be46646b
Nov 01 20:13:11.530473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.530438    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:11.530567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.530511    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:11.530607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.530577    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:11.936030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.935923    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" event={"ID":"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4","Type":"ContainerStarted","Data":"7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"}
Nov 01 20:13:11.936030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.936003    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" event={"ID":"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4","Type":"ContainerStarted","Data":"0c9eeb4dc0967c4c67351600f73d91d9547607debd36b3e9a954c4a6be46646b"}
Nov 01 20:13:11.957856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:11.957786    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" podStartSLOduration=1.957768525 podStartE2EDuration="1.957768525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:11.955884003 +0000 UTC m=+837.371516477" watchObservedRunningTime="2026-11-01 20:13:11.957768525 +0000 UTC m=+837.373401014"
Nov 01 20:13:12.131713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.129357    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:12.200622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.200588    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:12.217285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.217254    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:12.372416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.372385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:12.471044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.470927    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:12.482496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.482468    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"]
Nov 01 20:13:12.487607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.487583    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.494457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.494431    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"]
Nov 01 20:13:12.639147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.637736    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" podUID="4366a078-9273-4995-a29c-d2021ea006b5" containerName="payload-processing" containerID="cri-o://25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3" gracePeriod=25
Nov 01 20:13:12.656705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.654627    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.656705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.654937    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.656705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.654969    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.656705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.654997    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.700774 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:12.682088    6479 logging.go:55] [core] [Channel #677 SubChannel #678]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.70:50052", ServerName: "10.134.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.70:50052: connect: connection refused"
Nov 01 20:13:12.723923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.723819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:12.756590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.756548    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.756790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.756599    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.756790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.756634    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.756790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.756768    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.757368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.757343    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.757775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.757748    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.759790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.759771    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.765005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.764983    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.807912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.807866    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:12.940036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.940004    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:13:12.943753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.943716    6479 generic.go:356] "Generic (PLEG): container finished" podID="4366a078-9273-4995-a29c-d2021ea006b5" containerID="25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3" exitCode=0
Nov 01 20:13:12.944049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.944026    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"
Nov 01 20:13:12.944169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.944065    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" event={"ID":"4366a078-9273-4995-a29c-d2021ea006b5","Type":"ContainerDied","Data":"25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3"}
Nov 01 20:13:12.944169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.944121    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" event={"ID":"4366a078-9273-4995-a29c-d2021ea006b5","Type":"ContainerDied","Data":"1d6ecb7f6c580b4ba95852240a1eee5cb752435df3c7e78c72dbec36f66f6ac9"}
Nov 01 20:13:12.944302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.944151    6479 scope.go:122] "RemoveContainer" containerID="25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3"
Nov 01 20:13:12.944302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.944294    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:12.945362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.945347    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:12.969966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.963561    6479 scope.go:122] "RemoveContainer" containerID="25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3"
Nov 01 20:13:12.969966 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:13:12.963944    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3\": container with ID starting with 25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3 not found: ID does not exist" containerID="25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3"
Nov 01 20:13:12.969966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:12.963982    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3"} err="failed to get container status \"25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3\": rpc error: code = NotFound desc = could not find container \"25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3\": container with ID starting with 25600779e0051ff4afc3f17e01d4845f481c6a71d38583477a9e235aeefbdff3 not found: ID does not exist"
Nov 01 20:13:13.056073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.055975    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:13.066306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.066275    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials\") pod \"4366a078-9273-4995-a29c-d2021ea006b5\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") "
Nov 01 20:13:13.066449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.066368    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume\") pod \"4366a078-9273-4995-a29c-d2021ea006b5\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") "
Nov 01 20:13:13.066520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.066504    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume\") pod \"4366a078-9273-4995-a29c-d2021ea006b5\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") "
Nov 01 20:13:13.066642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.066627    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24\") pod \"4366a078-9273-4995-a29c-d2021ea006b5\" (UID: \"4366a078-9273-4995-a29c-d2021ea006b5\") "
Nov 01 20:13:13.066642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.066774    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume" pod "4366a078-9273-4995-a29c-d2021ea006b5" (UID: "4366a078-9273-4995-a29c-d2021ea006b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:13.066642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.066880    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume" pod "4366a078-9273-4995-a29c-d2021ea006b5" (UID: "4366a078-9273-4995-a29c-d2021ea006b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:13.067404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.067278    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:13.067503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.067482    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4366a078-9273-4995-a29c-d2021ea006b5-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:13.069165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.069136    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24" pod "4366a078-9273-4995-a29c-d2021ea006b5" (UID: "4366a078-9273-4995-a29c-d2021ea006b5"). InnerVolumeSpecName "kube-api-access-w5l24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:13.069497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.069466    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials" pod "4366a078-9273-4995-a29c-d2021ea006b5" (UID: "4366a078-9273-4995-a29c-d2021ea006b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:13.107657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.107620    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"]
Nov 01 20:13:13.109429 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:13.109390    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36f0805_dcf7_42be_a462_84beaeb0c7b0.slice/crio-1c7e7457d9ae71b27e5bee645038a6dfd08420aab91f3c9426ff2f733ffdd0db WatchSource:0}: Error finding container 1c7e7457d9ae71b27e5bee645038a6dfd08420aab91f3c9426ff2f733ffdd0db: Status 404 returned error can't find the container with id 1c7e7457d9ae71b27e5bee645038a6dfd08420aab91f3c9426ff2f733ffdd0db
Nov 01 20:13:13.154388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.154350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:13.168393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.168367    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5l24\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-kube-api-access-w5l24\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:13.168393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.168393    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4366a078-9273-4995-a29c-d2021ea006b5-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:13.370714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.369716    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"]
Nov 01 20:13:13.389673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.389420    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm"]
Nov 01 20:13:13.638751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.638673    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:13.681980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.681938    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-65796vsgm" podUID="4366a078-9273-4995-a29c-d2021ea006b5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.70:50052\" within 1s: context deadline exceeded"
Nov 01 20:13:13.738865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.738829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:13.957946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.957892    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp" event={"ID":"f36f0805-dcf7-42be-a462-84beaeb0c7b0","Type":"ContainerStarted","Data":"d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa"}
Nov 01 20:13:13.958329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.957971    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp" event={"ID":"f36f0805-dcf7-42be-a462-84beaeb0c7b0","Type":"ContainerStarted","Data":"1c7e7457d9ae71b27e5bee645038a6dfd08420aab91f3c9426ff2f733ffdd0db"}
Nov 01 20:13:13.958329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.958079    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:13.978137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:13.978079    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp" podStartSLOduration=1.9780292990000001 podStartE2EDuration="1.978029299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:13.97532765 +0000 UTC m=+839.390960122" watchObservedRunningTime="2026-11-01 20:13:13.978029299 +0000 UTC m=+839.393661773"
Nov 01 20:13:14.379099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.378973    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:14.449621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.449588    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"]
Nov 01 20:13:14.450197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.450184    6479 cpu_manager.go:414] "containerMap: removing container" podUID="4366a078-9273-4995-a29c-d2021ea006b5" containerName="payload-processing"
Nov 01 20:13:14.450246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.450199    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4366a078-9273-4995-a29c-d2021ea006b5" containerName="payload-processing"
Nov 01 20:13:14.450309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.450300    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="4366a078-9273-4995-a29c-d2021ea006b5" containerName="payload-processing"
Nov 01 20:13:14.453833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.453810    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.464795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.464770    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bccce1\""
Nov 01 20:13:14.465253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.465230    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"]
Nov 01 20:13:14.465577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.465552    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bccce1-dockercfg-v7d5z\""
Nov 01 20:13:14.485267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.485111    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:14.634727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.633425    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvtj\" (UniqueName: \"kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj\") pod \"payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.634727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.633574    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.664725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.662695    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:14.736724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.736688    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvtj\" (UniqueName: \"kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj\") pod \"payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.750704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.737099    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.751442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.738994    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.800037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.799997    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvtj\" (UniqueName: \"kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj\") pod \"payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:14.822687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.813480    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4366a078-9273-4995-a29c-d2021ea006b5" path="/var/lib/kubelet/pods/4366a078-9273-4995-a29c-d2021ea006b5/volumes"
Nov 01 20:13:14.846481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:14.845610    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:15.074837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.074806    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bccce1-dockercfg-v7d5z\""
Nov 01 20:13:15.082876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.082854    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:15.182981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.182908    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:15.247214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.247181    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"]
Nov 01 20:13:15.252622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.252592    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.259175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.259146    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"]
Nov 01 20:13:15.260592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.260568    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0ebb0cff-dockercfg-lf8fx\""
Nov 01 20:13:15.260924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.260906    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ebb0cff\""
Nov 01 20:13:15.264985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.264961    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.265205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.265178    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dcr\" (UniqueName: \"kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr\") pod \"payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.286723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.286696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:15.368040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.367952    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dcr\" (UniqueName: \"kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr\") pod \"payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.368485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.368459    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.369677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.369636    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.412664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.406289    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dcr\" (UniqueName: \"kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr\") pod \"payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.578173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.576907    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:15.604351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.604319    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"]
Nov 01 20:13:15.871319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.871293    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"]
Nov 01 20:13:15.872805 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:15.872666    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c39070f_962b_4b0a_853c_41e3c1af4d94.slice/crio-e0060468ad2b3c8636eb82470031535bcad475ce5c108b173d52993bd69858f5 WatchSource:0}: Error finding container e0060468ad2b3c8636eb82470031535bcad475ce5c108b173d52993bd69858f5: Status 404 returned error can't find the container with id e0060468ad2b3c8636eb82470031535bcad475ce5c108b173d52993bd69858f5
Nov 01 20:13:15.985056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.985019    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb" event={"ID":"0c39070f-962b-4b0a-853c-41e3c1af4d94","Type":"ContainerStarted","Data":"e0060468ad2b3c8636eb82470031535bcad475ce5c108b173d52993bd69858f5"}
Nov 01 20:13:15.986902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.986869    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" event={"ID":"d3644b62-073a-4e34-acc2-b0657b254966","Type":"ContainerStarted","Data":"57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861"}
Nov 01 20:13:15.987051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.986915    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" event={"ID":"d3644b62-073a-4e34-acc2-b0657b254966","Type":"ContainerStarted","Data":"4c1485449eead96e6121a89aba98e79f10823e83f8a9768418321dd22310f696"}
Nov 01 20:13:15.987728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:15.987709    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:16.000554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:16.000528    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:16.006469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:16.006427    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" podStartSLOduration=2.0064119 podStartE2EDuration="2.0064119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:16.00359723 +0000 UTC m=+841.419229704" watchObservedRunningTime="2026-11-01 20:13:16.0064119 +0000 UTC m=+841.422044372"
Nov 01 20:13:16.754084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:16.754039    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:16.995287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:16.995249    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb" event={"ID":"0c39070f-962b-4b0a-853c-41e3c1af4d94","Type":"ContainerStarted","Data":"8afdc3b9031c58217c5c299a090f5abcf0bfc70eb58f8c46af03e2ce9ea0d855"}
Nov 01 20:13:16.995512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:16.995337    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:17.017221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:17.017152    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb" podStartSLOduration=2.01713669 podStartE2EDuration="2.01713669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:17.01535904 +0000 UTC m=+842.430991525" watchObservedRunningTime="2026-11-01 20:13:17.01713669 +0000 UTC m=+842.432769162"
Nov 01 20:13:17.546325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:17.546281    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:18.151757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:18.151720    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:18.745612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:18.745578    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:19.344836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.344801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:19.915215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.915175    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"]
Nov 01 20:13:19.922324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.922296    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:19.928523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.928490    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"]
Nov 01 20:13:19.929347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.929323    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bccce1-serving-cert\""
Nov 01 20:13:19.971738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.971704    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:19.977210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.977179    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:19.977362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.977347    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:19.977531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.977504    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qk7s\" (UniqueName: \"kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:19.977708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:19.977685    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.071940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.071901    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:20.078783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.078752    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.078941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.078863    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qk7s\" (UniqueName: \"kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.078998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.078982    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.079058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.079045    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.079785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.079757    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.081478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.081449    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.081570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.081545    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.133669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.130846    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qk7s\" (UniqueName: \"kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s\") pod \"maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.246614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.246577    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:20.456728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.456660    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:20.531585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.531555    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"]
Nov 01 20:13:20.533709 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:20.533635    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39fa649_e41d_4ed0_9858_5024d1fc3a5b.slice/crio-6bd12d94daa387de62f8080495544be7d50af3d24897015beb8b6ed54458b6d8 WatchSource:0}: Error finding container 6bd12d94daa387de62f8080495544be7d50af3d24897015beb8b6ed54458b6d8: Status 404 returned error can't find the container with id 6bd12d94daa387de62f8080495544be7d50af3d24897015beb8b6ed54458b6d8
Nov 01 20:13:20.558890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.558834    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:20.658614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.658563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:20.880244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.877257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:20.970706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:20.970672    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:21.021670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:21.021606    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d" event={"ID":"e39fa649-e41d-4ed0-9858-5024d1fc3a5b","Type":"ContainerStarted","Data":"6bd12d94daa387de62f8080495544be7d50af3d24897015beb8b6ed54458b6d8"}
Nov 01 20:13:21.434571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:21.434538    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:21.582685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:21.582413    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:21.843749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:21.829264    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:21.918286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:21.918253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:22.019995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.019953    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:22.102721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.097523    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d" event={"ID":"e39fa649-e41d-4ed0-9858-5024d1fc3a5b","Type":"ContainerStarted","Data":"1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e"}
Nov 01 20:13:22.102721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.098611    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:22.236208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.236154    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d" podStartSLOduration=2.503604141 podStartE2EDuration="3.236139786s" totalImagesPullingTime="732.535645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:22.131008117 +0000 UTC m=+847.546640590" watchObservedRunningTime="2026-11-01 20:13:22.236139786 +0000 UTC m=+847.651772294"
Nov 01 20:13:22.236780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.236752    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"]
Nov 01 20:13:22.239514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.239493    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:22.263833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.263806    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"]
Nov 01 20:13:22.268338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.268319    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.275152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.274997    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-72be294cb636\""
Nov 01 20:13:22.275152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.275106    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"]
Nov 01 20:13:22.386128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.386034    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.386128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.386094    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwcr\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.386330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.386257    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.386412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.386391    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.488395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.488352    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.488768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.488742    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.488996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.488978    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.489108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.489092    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwcr\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.490129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.489978    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.490129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.490063    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.492130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.492109    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.497349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.497330    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwcr\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.602141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.600891    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:22.720683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.720200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:22.918347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:22.918323    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"]
Nov 01 20:13:22.919762 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:22.919734    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2473191_76d3_4ca9_a191_cb2fa7f46be1.slice/crio-e4ad9a02062ef1353e5e7b167f546974a868aeae6e58610539a2e827c8da199f WatchSource:0}: Error finding container e4ad9a02062ef1353e5e7b167f546974a868aeae6e58610539a2e827c8da199f: Status 404 returned error can't find the container with id e4ad9a02062ef1353e5e7b167f546974a868aeae6e58610539a2e827c8da199f
Nov 01 20:13:23.106632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.106585    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv" event={"ID":"d2473191-76d3-4ca9-a191-cb2fa7f46be1","Type":"ContainerStarted","Data":"d4377507f6b8e7db4302bd2202659bfca0bdac3cd114a2754ef1192d87f9ddd4"}
Nov 01 20:13:23.106826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.106682    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv" event={"ID":"d2473191-76d3-4ca9-a191-cb2fa7f46be1","Type":"ContainerStarted","Data":"e4ad9a02062ef1353e5e7b167f546974a868aeae6e58610539a2e827c8da199f"}
Nov 01 20:13:23.106826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.106732    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:23.128270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.128221    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv" podStartSLOduration=1.128205934 podStartE2EDuration="1.128205934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:23.127124821 +0000 UTC m=+848.542757300" watchObservedRunningTime="2026-11-01 20:13:23.128205934 +0000 UTC m=+848.543838407"
Nov 01 20:13:23.342181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.342151    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:23.840154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.840120    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:13:23.951233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:23.951197    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:24.674635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:24.674430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:25.262709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:25.261853    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:25.854028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:25.853984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:26.467762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:26.467727    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:27.076840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.076789    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:27.237711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.237630    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp" podUID="f36f0805-dcf7-42be-a462-84beaeb0c7b0" containerName="payload-processing" containerID="cri-o://d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa" gracePeriod=25
Nov 01 20:13:27.524215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.524182    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:27.604082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604047    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume\") pod \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") "
Nov 01 20:13:27.604276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604197    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials\") pod \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") "
Nov 01 20:13:27.604343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604285    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr\") pod \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") "
Nov 01 20:13:27.604403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604356    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume\") pod \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\" (UID: \"f36f0805-dcf7-42be-a462-84beaeb0c7b0\") "
Nov 01 20:13:27.604488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604423    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume" pod "f36f0805-dcf7-42be-a462-84beaeb0c7b0" (UID: "f36f0805-dcf7-42be-a462-84beaeb0c7b0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:27.604804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604775    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume" pod "f36f0805-dcf7-42be-a462-84beaeb0c7b0" (UID: "f36f0805-dcf7-42be-a462-84beaeb0c7b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:27.604975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604953    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:27.605040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.604983    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f36f0805-dcf7-42be-a462-84beaeb0c7b0-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:27.606365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.606310    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials" pod "f36f0805-dcf7-42be-a462-84beaeb0c7b0" (UID: "f36f0805-dcf7-42be-a462-84beaeb0c7b0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:27.606452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.606392    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr" pod "f36f0805-dcf7-42be-a462-84beaeb0c7b0" (UID: "f36f0805-dcf7-42be-a462-84beaeb0c7b0"). InnerVolumeSpecName "kube-api-access-nscjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:27.706064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.706027    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:27.706064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.706060    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/f36f0805-dcf7-42be-a462-84beaeb0c7b0-kube-api-access-nscjr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:27.855749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:27.855707    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:28.006718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.006686    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:28.006902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.006874    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:13:28.166532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.166496    6479 generic.go:356] "Generic (PLEG): container finished" podID="f36f0805-dcf7-42be-a462-84beaeb0c7b0" containerID="d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa" exitCode=0
Nov 01 20:13:28.166752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.166539    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"
Nov 01 20:13:28.166752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.166577    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp" event={"ID":"f36f0805-dcf7-42be-a462-84beaeb0c7b0","Type":"ContainerDied","Data":"d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa"}
Nov 01 20:13:28.166752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.166631    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp" event={"ID":"f36f0805-dcf7-42be-a462-84beaeb0c7b0","Type":"ContainerDied","Data":"1c7e7457d9ae71b27e5bee645038a6dfd08420aab91f3c9426ff2f733ffdd0db"}
Nov 01 20:13:28.166752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.166689    6479 scope.go:122] "RemoveContainer" containerID="d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa"
Nov 01 20:13:28.178520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.178479    6479 scope.go:122] "RemoveContainer" containerID="d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa"
Nov 01 20:13:28.178845 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:13:28.178799    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa\": container with ID starting with d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa not found: ID does not exist" containerID="d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa"
Nov 01 20:13:28.178934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.178854    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa"} err="failed to get container status \"d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa\": rpc error: code = NotFound desc = could not find container \"d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa\": container with ID starting with d0e06d1ea8f65dd0fcab93b14b880044b361d0d6d2fecd51ce841421a59eeafa not found: ID does not exist"
Nov 01 20:13:28.192149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.192121    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"]
Nov 01 20:13:28.195004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.194980    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5gvkfp"]
Nov 01 20:13:28.565897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.565864    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:28.711413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:28.711383    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36f0805-dcf7-42be-a462-84beaeb0c7b0" path="/var/lib/kubelet/pods/f36f0805-dcf7-42be-a462-84beaeb0c7b0/volumes"
Nov 01 20:13:29.118872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:29.118837    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:13:29.151738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:29.151697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:29.183711 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:13:29.182751    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:13:29.718947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:29.718900    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:30.427216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:30.427180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:31.159382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:31.159350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:31.770265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:31.770230    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:32.386314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:32.386274    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:32.976276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:32.976238    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:33.651776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:33.651733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:34.116169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:34.116140    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:13:34.278576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:34.278541    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:34.949700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:34.949639    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:35.716325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:35.716288    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:36.394229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:36.394193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:37.435352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:37.435316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:38.063698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:38.063166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:38.293521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:38.293480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:38.770712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:38.770676    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:39.313368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:39.313331    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:39.410541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:39.410494    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:39.574114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:39.574022    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:39.789722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:39.788619    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:39.879244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:39.879121    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:40.389574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:40.389534    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:40.485027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:40.484992    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:40.995693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:40.995631    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:41.089173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.089140    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"]
Nov 01 20:13:41.090127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.090097    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f36f0805-dcf7-42be-a462-84beaeb0c7b0" containerName="payload-processing"
Nov 01 20:13:41.090127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.090126    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36f0805-dcf7-42be-a462-84beaeb0c7b0" containerName="payload-processing"
Nov 01 20:13:41.090343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.090295    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36f0805-dcf7-42be-a462-84beaeb0c7b0" containerName="payload-processing"
Nov 01 20:13:41.094347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.094326    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"]
Nov 01 20:13:41.094473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.094414    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.100945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.100917    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-29b57c\""
Nov 01 20:13:41.100945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.100929    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"openshift-service-ca.crt\""
Nov 01 20:13:41.101129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.101023    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"kube-root-ca.crt\""
Nov 01 20:13:41.101129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.101028    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"routing-overlay\""
Nov 01 20:13:41.108174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.108143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:41.108281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.108229    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82nc\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.108360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.108279    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.108417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.108403    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.108473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.108436    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.141992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.141962    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"agc-cred-6da735d022c1\""
Nov 01 20:13:41.142164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.142066    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"payload-processing-external-model-e2e--80e6ed0f-dockercfg-9xw72\""
Nov 01 20:13:41.218160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.218109    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82nc\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.218455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.218434    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.218695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.218673    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.218835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.218809    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.244669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.220496    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.244669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.221609    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.257724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.254030    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82nc\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.311256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.311222    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.407669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.400426    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:41.542911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.454623    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:41.543564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.534328    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"]
Nov 01 20:13:41.659131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.655866    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"]
Nov 01 20:13:41.659772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.659747    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.693617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.688995    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bccce1\""
Nov 01 20:13:41.693617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.689212    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"routing-overlay\""
Nov 01 20:13:41.693617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.689339    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"kube-root-ca.crt\""
Nov 01 20:13:41.693617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.689416    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"openshift-service-ca.crt\""
Nov 01 20:13:41.693617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.692693    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-e2e--900891f9-dockercfg-ns4ld\""
Nov 01 20:13:41.693617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.692774    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-646b8d4cafa4\""
Nov 01 20:13:41.763461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.763414    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7gt\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.763776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.763751    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.763956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.763939    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.764129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.764113    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.867838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.865587    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7gt\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.867838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.865701    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.867838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.865771    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.867838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.865852    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.867838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.866786    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.891763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.891289    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.917684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.915573    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7gt\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:41.978670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:41.947104    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:42.043554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:42.043515    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:42.147351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:42.090618    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:42.626715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:42.626689    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"]
Nov 01 20:13:42.730560 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:13:42.730451    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8803588_0122_4d3b_9053_f0626f3a14cd.slice/crio-621c8618d4309f0da4d4df6f305ae4329ffa59a12caec967f09f5ed7786d7209 WatchSource:0}: Error finding container 621c8618d4309f0da4d4df6f305ae4329ffa59a12caec967f09f5ed7786d7209: Status 404 returned error can't find the container with id 621c8618d4309f0da4d4df6f305ae4329ffa59a12caec967f09f5ed7786d7209
Nov 01 20:13:42.739506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:42.739475    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"]
Nov 01 20:13:42.777589 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:42.777485    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:43.471670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.471486    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:43.638670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.638052    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk" event={"ID":"d8803588-0122-4d3b-9053-f0626f3a14cd","Type":"ContainerStarted","Data":"f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af"}
Nov 01 20:13:43.638670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.638129    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk" event={"ID":"d8803588-0122-4d3b-9053-f0626f3a14cd","Type":"ContainerStarted","Data":"621c8618d4309f0da4d4df6f305ae4329ffa59a12caec967f09f5ed7786d7209"}
Nov 01 20:13:43.661669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.651526    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:43.707688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.704848    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk" podStartSLOduration=2.7048250080000003 podStartE2EDuration="2.704825008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:43.703442417 +0000 UTC m=+869.119074892" watchObservedRunningTime="2026-11-01 20:13:43.704825008 +0000 UTC m=+869.120457482"
Nov 01 20:13:43.710325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.710299    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl" event={"ID":"c1aac181-5c09-4832-b088-0132afbecf31","Type":"ContainerStarted","Data":"ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0"}
Nov 01 20:13:43.710491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.710476    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl" event={"ID":"c1aac181-5c09-4832-b088-0132afbecf31","Type":"ContainerStarted","Data":"031eaa0ef1c79711f3d3196a71751e9e955cecc9b3f5dbd32ca7ee385d59b01a"}
Nov 01 20:13:43.711250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.711195    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:43.730270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:43.730160    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl" podStartSLOduration=2.730141673 podStartE2EDuration="2.730141673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:43.727210709 +0000 UTC m=+869.142843182" watchObservedRunningTime="2026-11-01 20:13:43.730141673 +0000 UTC m=+869.145774151"
Nov 01 20:13:44.241606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:44.241567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:44.840113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:44.840081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:45.474671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:45.470976    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:46.087671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:46.086970    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:46.743755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:46.743711    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:47.379223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:47.376203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:47.981514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:47.981446    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:48.645130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:48.645083    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:49.129845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.129803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:49.296720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.288051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:49.530018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.529983    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"]
Nov 01 20:13:49.613680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.611724    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:49.713360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.713327    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:49.736557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.736518    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"]
Nov 01 20:13:49.754481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.752158    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"]
Nov 01 20:13:49.795960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.795926    6479 generic.go:356] "Generic (PLEG): container finished" podID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerID="7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4" exitCode=0
Nov 01 20:13:49.796144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.796036    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" event={"ID":"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4","Type":"ContainerDied","Data":"7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"}
Nov 01 20:13:49.796449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.796433    6479 scope.go:122] "RemoveContainer" containerID="7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"
Nov 01 20:13:49.798535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.798501    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:49.798687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.798591    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:49.798687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.798641    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:49.832924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.832609    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:49.917008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:49.916956    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:50.283667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:50.278595    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:13:50.377252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:50.377198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:50.768804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:50.768125    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:50.813300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:50.813268    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" event={"ID":"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4","Type":"ContainerStarted","Data":"171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3"}
Nov 01 20:13:51.196995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:51.196958    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:51.197310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:51.197287    6479 patch_prober.go:29] interesting pod/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:13:51.197482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:51.197452    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 20:13:51.695801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:51.695762    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:52.196780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:52.196747    6479 patch_prober.go:29] interesting pod/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:13:52.197182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:52.196816    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 20:13:52.737270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:52.737233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:53.014038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.012464    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": read tcp 10.134.0.2:39352->10.134.0.71:15021: read: connection reset by peer" start-of-body=
Nov 01 20:13:53.014038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.012530    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": read tcp 10.134.0.2:39352->10.134.0.71:15021: read: connection reset by peer"
Nov 01 20:13:53.196508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.196475    6479 patch_prober.go:29] interesting pod/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:13:53.196711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.196550    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 20:13:53.571220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.571095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:53.890431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.890345    6479 generic.go:356] "Generic (PLEG): container finished" podID="9af32888-a7a6-444e-be15-22035c3373ac" containerID="ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64" exitCode=0
Nov 01 20:13:53.890595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.890578    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" event={"ID":"9af32888-a7a6-444e-be15-22035c3373ac","Type":"ContainerDied","Data":"ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"}
Nov 01 20:13:53.891117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.891096    6479 scope.go:122] "RemoveContainer" containerID="ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"
Nov 01 20:13:53.892961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.892925    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:53.893069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.893017    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:53.893112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:53.893065    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:13:54.196267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.196236    6479 patch_prober.go:29] interesting pod/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:13:54.196421 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.196294    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 20:13:54.431636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.431072    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:54.531044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.530953    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb" podUID="0c39070f-962b-4b0a-853c-41e3c1af4d94" containerName="payload-pre-processing" containerID="cri-o://8afdc3b9031c58217c5c299a090f5abcf0bfc70eb58f8c46af03e2ce9ea0d855" gracePeriod=25
Nov 01 20:13:54.579691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.579176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:54.677702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.677110    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:54.746667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.745733    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerName="payload-processing" containerID="cri-o://d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b" gracePeriod=25
Nov 01 20:13:54.770715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.757898    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" containerName="payload-pre-processing" containerID="cri-o://2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253" gracePeriod=25
Nov 01 20:13:54.873669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:54.872969    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:55.010305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.010214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:55.015563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.015427    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"]
Nov 01 20:13:55.015751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.015666    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy" containerID="cri-o://171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3" gracePeriod=30
Nov 01 20:13:55.017376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.017130    6479 generic.go:356] "Generic (PLEG): container finished" podID="0c39070f-962b-4b0a-853c-41e3c1af4d94" containerID="8afdc3b9031c58217c5c299a090f5abcf0bfc70eb58f8c46af03e2ce9ea0d855" exitCode=0
Nov 01 20:13:55.017376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.017245    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb" event={"ID":"0c39070f-962b-4b0a-853c-41e3c1af4d94","Type":"ContainerDied","Data":"8afdc3b9031c58217c5c299a090f5abcf0bfc70eb58f8c46af03e2ce9ea0d855"}
Nov 01 20:13:55.023052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.022246    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" event={"ID":"9af32888-a7a6-444e-be15-22035c3373ac","Type":"ContainerStarted","Data":"1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a"}
Nov 01 20:13:55.568912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.568193    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:13:55.575161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.574996    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 20:13:55.575161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.575049    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 20:13:55.638725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.638127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:55.640312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.640278    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:55.732669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.731452    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:55.732669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.731625    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr\" (UniqueName: \"kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr\") pod \"0c39070f-962b-4b0a-853c-41e3c1af4d94\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") "
Nov 01 20:13:55.732669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.731748    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume\") pod \"0c39070f-962b-4b0a-853c-41e3c1af4d94\" (UID: \"0c39070f-962b-4b0a-853c-41e3c1af4d94\") "
Nov 01 20:13:55.732669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.732312    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:13:55.733023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.732790    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume" pod "0c39070f-962b-4b0a-853c-41e3c1af4d94" (UID: "0c39070f-962b-4b0a-853c-41e3c1af4d94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:55.738942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.736350    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:13:55.739481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.739457    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr" pod "0c39070f-962b-4b0a-853c-41e3c1af4d94" (UID: "0c39070f-962b-4b0a-853c-41e3c1af4d94"). InnerVolumeSpecName "kube-api-access-v8dcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:55.828697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.828264    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:55.835683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.834069    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dcr\" (UniqueName: \"kubernetes.io/projected/0c39070f-962b-4b0a-853c-41e3c1af4d94-kube-api-access-v8dcr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:55.835683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.834091    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c39070f-962b-4b0a-853c-41e3c1af4d94-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:55.957671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:55.956493    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:56.077675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.077494    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"
Nov 01 20:13:56.084673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.079828    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb" event={"ID":"0c39070f-962b-4b0a-853c-41e3c1af4d94","Type":"ContainerDied","Data":"e0060468ad2b3c8636eb82470031535bcad475ce5c108b173d52993bd69858f5"}
Nov 01 20:13:56.084673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.079898    6479 scope.go:122] "RemoveContainer" containerID="8afdc3b9031c58217c5c299a090f5abcf0bfc70eb58f8c46af03e2ce9ea0d855"
Nov 01 20:13:56.265689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.265549    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:56.279706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.277244    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"]
Nov 01 20:13:56.292659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.292590    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ebb0cff-774dd8786c-7xthb"]
Nov 01 20:13:56.432207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.432159    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:56.545357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.545223    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:56.546100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.546076    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 20:13:56.546283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.546261    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 20:13:56.821753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.798709    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c39070f-962b-4b0a-853c-41e3c1af4d94" path="/var/lib/kubelet/pods/0c39070f-962b-4b0a-853c-41e3c1af4d94/volumes"
Nov 01 20:13:56.824447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:56.824417    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:57.445058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:57.445019    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:57.557277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:57.550254    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 20:13:57.557277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:57.550381    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 20:13:57.944338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:57.944316    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:13:58.048751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:58.048720    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:58.546443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:58.546409    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 20:13:58.546609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:58.546491    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 20:13:58.690298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:58.690263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:59.206667 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:13:59.206616    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:13:59.220214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.220182    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:59.246708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.246665    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:13:59.246708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.246682    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:13:59.317610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.317575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:59.419291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.419255    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:59.557759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.557639    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 20:13:59.558042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.557998    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 20:13:59.807170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.807126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:59.893805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.893715    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:13:59.894549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.894526    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"]
Nov 01 20:13:59.895235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.895215    6479 cpu_manager.go:414] "containerMap: removing container" podUID="0c39070f-962b-4b0a-853c-41e3c1af4d94" containerName="payload-pre-processing"
Nov 01 20:13:59.895337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.895237    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c39070f-962b-4b0a-853c-41e3c1af4d94" containerName="payload-pre-processing"
Nov 01 20:13:59.895444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.895429    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c39070f-962b-4b0a-853c-41e3c1af4d94" containerName="payload-pre-processing"
Nov 01 20:13:59.898893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.898857    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:13:59.907739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.906814    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-91f1523a\""
Nov 01 20:13:59.908403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.908372    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"]
Nov 01 20:13:59.908933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.908910    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-91f1523a-dockercfg-ttr6k\""
Nov 01 20:13:59.934416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.934381    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"]
Nov 01 20:13:59.939475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.939454    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:13:59.947392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:13:59.947361    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"]
Nov 01 20:14:00.049701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.048691    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptzz\" (UniqueName: \"kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz\") pod \"payload-processing-e2e-migrate-91f1523a-899695877-8wlls\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") " pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.049701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.048756    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-91f1523a-899695877-8wlls\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") " pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.049701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.048941    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.049701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.049116    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4xd\" (UniqueName: \"kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd\") pod \"payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.173176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.161690    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4xd\" (UniqueName: \"kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd\") pod \"payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.173176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.161793    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptzz\" (UniqueName: \"kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz\") pod \"payload-processing-e2e-migrate-91f1523a-899695877-8wlls\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") " pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.173176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.161831    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-91f1523a-899695877-8wlls\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") " pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.173176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.161972    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.196757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.194518    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.196757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.195249    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-91f1523a-899695877-8wlls\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") " pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.309695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.309664    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptzz\" (UniqueName: \"kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz\") pod \"payload-processing-e2e-migrate-91f1523a-899695877-8wlls\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") " pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.310346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.310323    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4xd\" (UniqueName: \"kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd\") pod \"payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.489321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.489281    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:00.517426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.517387    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:00.547753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.546788    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 20:14:00.547753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.546867    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 20:14:00.551779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:00.551709    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:01.021503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.021476    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"]
Nov 01 20:14:01.023997 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:01.023963    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf324287a_e272_46f4_8277_7e7b8ba6bf9e.slice/crio-2c27ea4a78f6a78e7011fb171b60bb48ab2d2b48e74b28ecac1ccf30a1b55057 WatchSource:0}: Error finding container 2c27ea4a78f6a78e7011fb171b60bb48ab2d2b48e74b28ecac1ccf30a1b55057: Status 404 returned error can't find the container with id 2c27ea4a78f6a78e7011fb171b60bb48ab2d2b48e74b28ecac1ccf30a1b55057
Nov 01 20:14:01.086280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.086241    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"]
Nov 01 20:14:01.089246 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:01.089214    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccec9bed_3733_4ea5_bbe3_5ef27cd690d5.slice/crio-eb8a00aab5fac614843622b6295a33a42a92656b60794c586a84dc7db89f4299 WatchSource:0}: Error finding container eb8a00aab5fac614843622b6295a33a42a92656b60794c586a84dc7db89f4299: Status 404 returned error can't find the container with id eb8a00aab5fac614843622b6295a33a42a92656b60794c586a84dc7db89f4299
Nov 01 20:14:01.276673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.271514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:01.383480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.383367    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9" event={"ID":"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5","Type":"ContainerStarted","Data":"27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64"}
Nov 01 20:14:01.384074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.384035    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9" event={"ID":"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5","Type":"ContainerStarted","Data":"eb8a00aab5fac614843622b6295a33a42a92656b60794c586a84dc7db89f4299"}
Nov 01 20:14:01.385278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.385257    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:01.405947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.405911    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls" event={"ID":"f324287a-e272-46f4-8277-7e7b8ba6bf9e","Type":"ContainerStarted","Data":"9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250"}
Nov 01 20:14:01.407523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.407263    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls" event={"ID":"f324287a-e272-46f4-8277-7e7b8ba6bf9e","Type":"ContainerStarted","Data":"2c27ea4a78f6a78e7011fb171b60bb48ab2d2b48e74b28ecac1ccf30a1b55057"}
Nov 01 20:14:01.407523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.407298    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:01.413698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.412502    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9" podStartSLOduration=2.412487511 podStartE2EDuration="2.412487511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:01.405027989 +0000 UTC m=+886.820660484" watchObservedRunningTime="2026-11-01 20:14:01.412487511 +0000 UTC m=+886.828119984"
Nov 01 20:14:01.426994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.426947    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls" podStartSLOduration=2.426934316 podStartE2EDuration="2.426934316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:01.423999245 +0000 UTC m=+886.839631731" watchObservedRunningTime="2026-11-01 20:14:01.426934316 +0000 UTC m=+886.842566822"
Nov 01 20:14:01.763314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.763282    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:14:01.763895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.763859    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:14:01.764829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:01.764793    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:14:02.041458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.041377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:02.200900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.200865    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:02.607978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.607949    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:14:02.653335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.653287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743370    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743475    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743550    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743583    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743609    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743733    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743801    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743834    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.743861    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token\") pod \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\" (UID: \"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4\") "
Nov 01 20:14:02.756679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.755941    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:02.772237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.761102    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:02.772237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.769996    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:02.772237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.770694    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:02.772237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.770747    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:02.790898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.790395    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:02.790898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.790569    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:14:02.790898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.790602    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "kube-api-access-ld5g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:02.792593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.792561    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token" pod "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" (UID: "66ad8539-02cd-4b6e-8d7a-d8bf72644ab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:02.846159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846112    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846159    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5g8\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-kube-api-access-ld5g8\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846175    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846188    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846201    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846214    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846226    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846240    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:02.846375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:02.846252    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:03.113369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.113266    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"]
Nov 01 20:14:03.114631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.114594    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" podUID="01e427c6-3721-46de-bd53-58ff20a46d5b" containerName="maas-api" containerID="cri-o://4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708" gracePeriod=30
Nov 01 20:14:03.178390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.178355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:03.272908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.272871    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" podUID="01e427c6-3721-46de-bd53-58ff20a46d5b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.62:8443/health\": dial tcp 10.134.0.62:8443: connect: connection refused"
Nov 01 20:14:03.368725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.358970    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:03.465678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.462186    6479 generic.go:356] "Generic (PLEG): container finished" podID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerID="171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3" exitCode=0
Nov 01 20:14:03.476753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.466098    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" event={"ID":"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4","Type":"ContainerDied","Data":"171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3"}
Nov 01 20:14:03.476957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.476935    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9" event={"ID":"66ad8539-02cd-4b6e-8d7a-d8bf72644ab4","Type":"ContainerDied","Data":"0c9eeb4dc0967c4c67351600f73d91d9547607debd36b3e9a954c4a6be46646b"}
Nov 01 20:14:03.482899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.481809    6479 scope.go:122] "RemoveContainer" containerID="171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3"
Nov 01 20:14:03.483196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.472781    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"
Nov 01 20:14:03.533604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.533211    6479 scope.go:122] "RemoveContainer" containerID="7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"
Nov 01 20:14:03.577801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.572140    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"]
Nov 01 20:14:03.578148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.578125    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ebb0cff-openshift-default-55464d4cbf-n4bk9"]
Nov 01 20:14:03.590955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.573226    6479 scope.go:122] "RemoveContainer" containerID="171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3"
Nov 01 20:14:03.590955 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:03.579364    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3\": container with ID starting with 171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3 not found: ID does not exist" containerID="171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3"
Nov 01 20:14:03.590955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.579404    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3"} err="failed to get container status \"171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3\": rpc error: code = NotFound desc = could not find container \"171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3\": container with ID starting with 171f16365f33ed4681ff71e634bc53f0fdc51e17616d68f1daeb17c177fb7af3 not found: ID does not exist"
Nov 01 20:14:03.590955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.579443    6479 scope.go:122] "RemoveContainer" containerID="7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"
Nov 01 20:14:03.590955 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:03.579817    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4\": container with ID starting with 7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4 not found: ID does not exist" containerID="7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"
Nov 01 20:14:03.590955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.579847    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4"} err="failed to get container status \"7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4\": rpc error: code = NotFound desc = could not find container \"7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4\": container with ID starting with 7d68bdefe1d68a250f9d2c829635bd8a65532d64159ac27783c42fa9d4fe13a4 not found: ID does not exist"
Nov 01 20:14:03.822523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.822497    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:14:03.884702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.884637    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms\" (UniqueName: \"kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms\") pod \"01e427c6-3721-46de-bd53-58ff20a46d5b\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") "
Nov 01 20:14:03.884916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.884750    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls\") pod \"01e427c6-3721-46de-bd53-58ff20a46d5b\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") "
Nov 01 20:14:03.884916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.884884    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca\") pod \"01e427c6-3721-46de-bd53-58ff20a46d5b\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") "
Nov 01 20:14:03.885047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.884942    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs\") pod \"01e427c6-3721-46de-bd53-58ff20a46d5b\" (UID: \"01e427c6-3721-46de-bd53-58ff20a46d5b\") "
Nov 01 20:14:03.885719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.885459    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca" pod "01e427c6-3721-46de-bd53-58ff20a46d5b" (UID: "01e427c6-3721-46de-bd53-58ff20a46d5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:03.885719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.885704    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e427c6-3721-46de-bd53-58ff20a46d5b-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:03.887141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.887101    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls" pod "01e427c6-3721-46de-bd53-58ff20a46d5b" (UID: "01e427c6-3721-46de-bd53-58ff20a46d5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:03.887371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.887339    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms" pod "01e427c6-3721-46de-bd53-58ff20a46d5b" (UID: "01e427c6-3721-46de-bd53-58ff20a46d5b"). InnerVolumeSpecName "kube-api-access-rt8ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:03.887722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.887701    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs" pod "01e427c6-3721-46de-bd53-58ff20a46d5b" (UID: "01e427c6-3721-46de-bd53-58ff20a46d5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:03.986725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.986630    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8ms\" (UniqueName: \"kubernetes.io/projected/01e427c6-3721-46de-bd53-58ff20a46d5b-kube-api-access-rt8ms\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:03.986725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.986675    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:03.986725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:03.986688    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e427c6-3721-46de-bd53-58ff20a46d5b-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:04.041491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.041451    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:04.484203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.484166    6479 generic.go:356] "Generic (PLEG): container finished" podID="01e427c6-3721-46de-bd53-58ff20a46d5b" containerID="4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708" exitCode=0
Nov 01 20:14:04.484388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.484247    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"
Nov 01 20:14:04.484388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.484258    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" event={"ID":"01e427c6-3721-46de-bd53-58ff20a46d5b","Type":"ContainerDied","Data":"4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708"}
Nov 01 20:14:04.484388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.484330    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2" event={"ID":"01e427c6-3721-46de-bd53-58ff20a46d5b","Type":"ContainerDied","Data":"2881d9207c6a85bdce42b7f6c8659e9ad17f32bf3de535f88a2d62ac2b1e77a2"}
Nov 01 20:14:04.484388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.484351    6479 scope.go:122] "RemoveContainer" containerID="4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708"
Nov 01 20:14:04.498913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.498881    6479 scope.go:122] "RemoveContainer" containerID="4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708"
Nov 01 20:14:04.499483 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:04.499453    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708\": container with ID starting with 4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708 not found: ID does not exist" containerID="4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708"
Nov 01 20:14:04.499572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.499496    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708"} err="failed to get container status \"4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708\": rpc error: code = NotFound desc = could not find container \"4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708\": container with ID starting with 4b273f49527cc7e8c6aef2cc337f0b2366fead88cac74bd9cb257e81b25d5708 not found: ID does not exist"
Nov 01 20:14:04.510682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.510609    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"]
Nov 01 20:14:04.513727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.513696    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-638f19-6d88dd97b6-kqbc2"]
Nov 01 20:14:04.641972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.641937    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:04.723782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.722338    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e427c6-3721-46de-bd53-58ff20a46d5b" path="/var/lib/kubelet/pods/01e427c6-3721-46de-bd53-58ff20a46d5b/volumes"
Nov 01 20:14:04.723782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:04.723090    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" path="/var/lib/kubelet/pods/66ad8539-02cd-4b6e-8d7a-d8bf72644ab4/volumes"
Nov 01 20:14:05.321487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.321452    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"]
Nov 01 20:14:05.322532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322504    6479 cpu_manager.go:414] "containerMap: removing container" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy"
Nov 01 20:14:05.322532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322532    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy"
Nov 01 20:14:05.322748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322574    6479 cpu_manager.go:414] "containerMap: removing container" podUID="01e427c6-3721-46de-bd53-58ff20a46d5b" containerName="maas-api"
Nov 01 20:14:05.322748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322583    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e427c6-3721-46de-bd53-58ff20a46d5b" containerName="maas-api"
Nov 01 20:14:05.322748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322611    6479 cpu_manager.go:414] "containerMap: removing container" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy"
Nov 01 20:14:05.322748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322619    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy"
Nov 01 20:14:05.322952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322774    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy"
Nov 01 20:14:05.322952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.322795    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e427c6-3721-46de-bd53-58ff20a46d5b" containerName="maas-api"
Nov 01 20:14:05.328689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.328634    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.337125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.333461    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-91f1523a-serving-cert\""
Nov 01 20:14:05.337125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.336067    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"]
Nov 01 20:14:05.354778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.354736    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:05.420896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.420865    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84n2\" (UniqueName: \"kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.421090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.420913    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.421151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.421101    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.421240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.421224    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.465679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.460482    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:05.524585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.524539    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.525154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.525125    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84n2\" (UniqueName: \"kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.525307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.525292    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.525553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.525534    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.535497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.526625    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.545715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.544939    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.548874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.546431    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.555605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.555572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:05.555938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.555622    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84n2\" (UniqueName: \"kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2\") pod \"maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:05.652806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:05.652705    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:06.006551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.006520    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"]
Nov 01 20:14:06.009058 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:06.009030    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2bf228_14b5_4d15_b76f_2b1b2f3a0183.slice/crio-7e9c85cca2a9a49308f2ee4ec70295e5fa2e1b7a1a722e2c5e4853d21f4c8ddd WatchSource:0}: Error finding container 7e9c85cca2a9a49308f2ee4ec70295e5fa2e1b7a1a722e2c5e4853d21f4c8ddd: Status 404 returned error can't find the container with id 7e9c85cca2a9a49308f2ee4ec70295e5fa2e1b7a1a722e2c5e4853d21f4c8ddd
Nov 01 20:14:06.096424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.096391    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:06.561092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.561060    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk" event={"ID":"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183","Type":"ContainerStarted","Data":"f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620"}
Nov 01 20:14:06.561441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.561116    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk" event={"ID":"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183","Type":"ContainerStarted","Data":"7e9c85cca2a9a49308f2ee4ec70295e5fa2e1b7a1a722e2c5e4853d21f4c8ddd"}
Nov 01 20:14:06.561441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.561143    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:06.579201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.579148    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk" podStartSLOduration=1.159180621 podStartE2EDuration="1.579133975s" totalImagesPullingTime="419.953354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:06.577504984 +0000 UTC m=+891.993137471" watchObservedRunningTime="2026-11-01 20:14:06.579133975 +0000 UTC m=+891.994766453"
Nov 01 20:14:06.965848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:06.965813    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:07.564307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:07.564262    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:08.007517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.007482    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:08.157779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.157740    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:08.266437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.266345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:08.452198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.452163    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:08.774725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.761824    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"]
Nov 01 20:14:08.774725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.762097    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy" containerID="cri-o://bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68" gracePeriod=30
Nov 01 20:14:08.802169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:08.802137    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:09.234953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:09.234920    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:09.245852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:09.245812    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:14:09.245985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:09.245852    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:14:09.846108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:09.846072    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:10.552911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:10.552874    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:11.155378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:11.155339    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:11.754027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:11.753983    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:12.341474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:12.341425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:12.446671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:12.444421    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:12.577572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:12.577544    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:13.216065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:13.216029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:13.480191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:13.480110    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:14.049544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.049515    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:14.058025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.058002    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:14:14.163232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.163193    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.163543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.163522    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.163684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.163666    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.163808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.163790    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.163933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.163918    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.164081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.164067    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.164230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.164218    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.171482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.171451    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.172032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.171880    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td\") pod \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\" (UID: \"405d10b4-c73c-461e-a9ac-0c17bb03fc3f\") "
Nov 01 20:14:14.174074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.173767    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.174523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.174502    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.175491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.175470    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:14.175866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.175848    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.176144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.176125    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.180189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.180152    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:14:14.229668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.219108    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.230347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.230115    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:14.237327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.236792    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td" pod "405d10b4-c73c-461e-a9ac-0c17bb03fc3f" (UID: "405d10b4-c73c-461e-a9ac-0c17bb03fc3f"). InnerVolumeSpecName "kube-api-access-tf6td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:14.238132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.237991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274325    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274350    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274361    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274371    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274381    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274390    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274400    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274409    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6td\" (UniqueName: \"kubernetes.io/projected/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-kube-api-access-tf6td\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.276237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.274433    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d10b4-c73c-461e-a9ac-0c17bb03fc3f-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.668360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.668327    6479 generic.go:356] "Generic (PLEG): container finished" podID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerID="bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68" exitCode=0
Nov 01 20:14:14.668836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.668383    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"
Nov 01 20:14:14.668836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.668414    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" event={"ID":"405d10b4-c73c-461e-a9ac-0c17bb03fc3f","Type":"ContainerDied","Data":"bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68"}
Nov 01 20:14:14.668836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.668482    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc" event={"ID":"405d10b4-c73c-461e-a9ac-0c17bb03fc3f","Type":"ContainerDied","Data":"2919cebfbff3c2c3b697cba88c081d811cc14a1e0318ced7f774021a6289b977"}
Nov 01 20:14:14.668836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.668513    6479 scope.go:122] "RemoveContainer" containerID="bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68"
Nov 01 20:14:14.680730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.680710    6479 scope.go:122] "RemoveContainer" containerID="9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"
Nov 01 20:14:14.707952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.707912    6479 scope.go:122] "RemoveContainer" containerID="bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68"
Nov 01 20:14:14.712143 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:14.712112    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68\": container with ID starting with bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68 not found: ID does not exist" containerID="bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68"
Nov 01 20:14:14.712260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.712157    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68"} err="failed to get container status \"bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68\": rpc error: code = NotFound desc = could not find container \"bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68\": container with ID starting with bbdaa3e1c6cc7229f61dd963a644d57919f2d886802d71fe0e9fde34ab430b68 not found: ID does not exist"
Nov 01 20:14:14.712260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.712183    6479 scope.go:122] "RemoveContainer" containerID="9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"
Nov 01 20:14:14.712672 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:14.712629    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c\": container with ID starting with 9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c not found: ID does not exist" containerID="9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"
Nov 01 20:14:14.712786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.712681    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c"} err="failed to get container status \"9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c\": rpc error: code = NotFound desc = could not find container \"9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c\": container with ID starting with 9454e2f7f4da819da241bbb351fa821203a5ac6013d727c5c109051575604c1c not found: ID does not exist"
Nov 01 20:14:14.717601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.717578    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"]
Nov 01 20:14:14.717719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.717614    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-frtbc"]
Nov 01 20:14:14.781824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:14.781781    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:15.032459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.032430    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:14:15.088096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.088058    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456\" (UniqueName: \"kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456\") pod \"7ee6f741-9e37-48bc-a647-0620e3cd2362\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") "
Nov 01 20:14:15.088295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.088167    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume\") pod \"7ee6f741-9e37-48bc-a647-0620e3cd2362\" (UID: \"7ee6f741-9e37-48bc-a647-0620e3cd2362\") "
Nov 01 20:14:15.088824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.088795    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume" pod "7ee6f741-9e37-48bc-a647-0620e3cd2362" (UID: "7ee6f741-9e37-48bc-a647-0620e3cd2362"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:15.099702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.091782    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456" pod "7ee6f741-9e37-48bc-a647-0620e3cd2362" (UID: "7ee6f741-9e37-48bc-a647-0620e3cd2362"). InnerVolumeSpecName "kube-api-access-rv456". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:15.117248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.117223    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:14:15.189375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.189323    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume\") pod \"8347908c-6d47-4167-9621-462e8f9c4d00\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") "
Nov 01 20:14:15.189613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.189449    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb\" (UniqueName: \"kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb\") pod \"8347908c-6d47-4167-9621-462e8f9c4d00\" (UID: \"8347908c-6d47-4167-9621-462e8f9c4d00\") "
Nov 01 20:14:15.189883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.189856    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume" pod "8347908c-6d47-4167-9621-462e8f9c4d00" (UID: "8347908c-6d47-4167-9621-462e8f9c4d00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:15.190159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.190136    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv456\" (UniqueName: \"kubernetes.io/projected/7ee6f741-9e37-48bc-a647-0620e3cd2362-kube-api-access-rv456\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:15.190242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.190164    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6f741-9e37-48bc-a647-0620e3cd2362-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:15.190242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.190177    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8347908c-6d47-4167-9621-462e8f9c4d00-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:15.191870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.191804    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb" pod "8347908c-6d47-4167-9621-462e8f9c4d00" (UID: "8347908c-6d47-4167-9621-462e8f9c4d00"). InnerVolumeSpecName "kube-api-access-gmmjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:15.291343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.291304    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmjb\" (UniqueName: \"kubernetes.io/projected/8347908c-6d47-4167-9621-462e8f9c4d00-kube-api-access-gmmjb\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:15.573805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.573775    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:15.724986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.724952    6479 generic.go:356] "Generic (PLEG): container finished" podID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerID="d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b" exitCode=0
Nov 01 20:14:15.731620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.728845    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" event={"ID":"7ee6f741-9e37-48bc-a647-0620e3cd2362","Type":"ContainerDied","Data":"d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b"}
Nov 01 20:14:15.731620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.728919    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd" event={"ID":"7ee6f741-9e37-48bc-a647-0620e3cd2362","Type":"ContainerDied","Data":"3c3d6d77c923c4d437b1be8e6f0b359d13829e4f7845ae9c11f4981b92d4e822"}
Nov 01 20:14:15.731620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.728960    6479 scope.go:122] "RemoveContainer" containerID="d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b"
Nov 01 20:14:15.731620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.729112    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"
Nov 01 20:14:15.739696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.739637    6479 generic.go:356] "Generic (PLEG): container finished" podID="8347908c-6d47-4167-9621-462e8f9c4d00" containerID="2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253" exitCode=0
Nov 01 20:14:15.739844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.739720    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"
Nov 01 20:14:15.739844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.739759    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" event={"ID":"8347908c-6d47-4167-9621-462e8f9c4d00","Type":"ContainerDied","Data":"2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253"}
Nov 01 20:14:15.739844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.739835    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl" event={"ID":"8347908c-6d47-4167-9621-462e8f9c4d00","Type":"ContainerDied","Data":"dc2dd73b56f1c024bd0bf97b3e05e6f2c179f9e38a1e266fbf5f517ae67edd11"}
Nov 01 20:14:15.748019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.747996    6479 scope.go:122] "RemoveContainer" containerID="d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b"
Nov 01 20:14:15.748285 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:15.748267    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b\": container with ID starting with d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b not found: ID does not exist" containerID="d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b"
Nov 01 20:14:15.748341 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.748294    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b"} err="failed to get container status \"d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b\": rpc error: code = NotFound desc = could not find container \"d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b\": container with ID starting with d5f325be343477f55312ae96d10529073cb498fef0fd84c850d7ffaecfd13b3b not found: ID does not exist"
Nov 01 20:14:15.748341 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.748312    6479 scope.go:122] "RemoveContainer" containerID="2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253"
Nov 01 20:14:15.758737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.758712    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"]
Nov 01 20:14:15.761841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.761795    6479 scope.go:122] "RemoveContainer" containerID="2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253"
Nov 01 20:14:15.762122 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:15.762100    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253\": container with ID starting with 2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253 not found: ID does not exist" containerID="2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253"
Nov 01 20:14:15.762188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.762132    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253"} err="failed to get container status \"2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253\": rpc error: code = NotFound desc = could not find container \"2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253\": container with ID starting with 2a0b0bc4157c445db2f59cdd5bac700f74084d911b54a872896cf87f888ec253 not found: ID does not exist"
Nov 01 20:14:15.762490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.762470    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-638f19-55f6fff6f7-2c7dd"]
Nov 01 20:14:15.773537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.773512    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"]
Nov 01 20:14:15.777558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:15.777538    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-638f19-749f6b8d8c-8gmwl"]
Nov 01 20:14:16.286117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:16.286081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:16.708558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:16.708524    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" path="/var/lib/kubelet/pods/405d10b4-c73c-461e-a9ac-0c17bb03fc3f/volumes"
Nov 01 20:14:16.709013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:16.708999    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" path="/var/lib/kubelet/pods/7ee6f741-9e37-48bc-a647-0620e3cd2362/volumes"
Nov 01 20:14:16.709357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:16.709345    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" path="/var/lib/kubelet/pods/8347908c-6d47-4167-9621-462e8f9c4d00/volumes"
Nov 01 20:14:16.979555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:16.979497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:17.669613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:17.669564    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:18.370161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:18.370124    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:19.307670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:19.299075    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:19.857193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:19.857149    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:20.479917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:20.479872    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:20.574838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:20.574798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:20.769718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:20.768318    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:21.683542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:21.683500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:21.872538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:21.872491    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:22.252037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:22.251995    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:22.947400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:22.947358    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:23.540241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:23.540210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:24.237436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:24.237400    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:24.278422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:24.278387    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"]
Nov 01 20:14:24.290909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:24.290870    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"]
Nov 01 20:14:24.339146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:24.339105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:24.445076 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:24.445041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:24.954182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:24.954100    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:25.589272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:25.589231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.040872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.040828    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"]
Nov 01 20:14:26.041179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.041147    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx" podUID="7e6e32a5-a0a1-4c2f-9708-66361456e02d" containerName="maas-api" containerID="cri-o://bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe" gracePeriod=30
Nov 01 20:14:26.090786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.090747    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.162871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.162836    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"]
Nov 01 20:14:26.163145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.163113    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk" podUID="2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" containerName="maas-api" containerID="cri-o://f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620" gracePeriod=30
Nov 01 20:14:26.219908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.219868    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.455078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.453942    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.584372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.584248    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.647289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.646961    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:14:26.674361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.674330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.702595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.702556    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca\") pod \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") "
Nov 01 20:14:26.702789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.702702    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls\") pod \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") "
Nov 01 20:14:26.702834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.702815    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs\") pod \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") "
Nov 01 20:14:26.702948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.702932    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s\" (UniqueName: \"kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s\") pod \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\" (UID: \"7e6e32a5-a0a1-4c2f-9708-66361456e02d\") "
Nov 01 20:14:26.703806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.703773    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca" pod "7e6e32a5-a0a1-4c2f-9708-66361456e02d" (UID: "7e6e32a5-a0a1-4c2f-9708-66361456e02d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:26.711380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.711335    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s" pod "7e6e32a5-a0a1-4c2f-9708-66361456e02d" (UID: "7e6e32a5-a0a1-4c2f-9708-66361456e02d"). InnerVolumeSpecName "kube-api-access-x674s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:26.711721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.711697    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs" pod "7e6e32a5-a0a1-4c2f-9708-66361456e02d" (UID: "7e6e32a5-a0a1-4c2f-9708-66361456e02d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:26.716477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.716429    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls" pod "7e6e32a5-a0a1-4c2f-9708-66361456e02d" (UID: "7e6e32a5-a0a1-4c2f-9708-66361456e02d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:26.734683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.734128    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:26.770828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.770781    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.804177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.804136    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls\") pod \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") "
Nov 01 20:14:26.804358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.804206    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca\") pod \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") "
Nov 01 20:14:26.804358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.804317    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2\" (UniqueName: \"kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2\") pod \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") "
Nov 01 20:14:26.804688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.804639    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca" pod "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" (UID: "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:26.804795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.804736    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs\") pod \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\" (UID: \"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183\") "
Nov 01 20:14:26.805357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.805332    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.805470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.805363    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x674s\" (UniqueName: \"kubernetes.io/projected/7e6e32a5-a0a1-4c2f-9708-66361456e02d-kube-api-access-x674s\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.805470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.805380    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.805470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.805395    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6e32a5-a0a1-4c2f-9708-66361456e02d-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.805470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.805410    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6e32a5-a0a1-4c2f-9708-66361456e02d-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.807558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.807535    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs" pod "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" (UID: "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:26.808107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.807917    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2" pod "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" (UID: "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183"). InnerVolumeSpecName "kube-api-access-g84n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:26.810077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.810051    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls" pod "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" (UID: "2f2bf228-14b5-4d15-b76f-2b1b2f3a0183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:26.845060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.845023    6479 generic.go:356] "Generic (PLEG): container finished" podID="2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" containerID="f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620" exitCode=0
Nov 01 20:14:26.845216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.845073    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"
Nov 01 20:14:26.845216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.845102    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk" event={"ID":"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183","Type":"ContainerDied","Data":"f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620"}
Nov 01 20:14:26.845216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.845176    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk" event={"ID":"2f2bf228-14b5-4d15-b76f-2b1b2f3a0183","Type":"ContainerDied","Data":"7e9c85cca2a9a49308f2ee4ec70295e5fa2e1b7a1a722e2c5e4853d21f4c8ddd"}
Nov 01 20:14:26.845216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.845211    6479 scope.go:122] "RemoveContainer" containerID="f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620"
Nov 01 20:14:26.846596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.846570    6479 generic.go:356] "Generic (PLEG): container finished" podID="7e6e32a5-a0a1-4c2f-9708-66361456e02d" containerID="bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe" exitCode=0
Nov 01 20:14:26.846742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.846618    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"
Nov 01 20:14:26.846742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.846660    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx" event={"ID":"7e6e32a5-a0a1-4c2f-9708-66361456e02d","Type":"ContainerDied","Data":"bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe"}
Nov 01 20:14:26.846742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.846722    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx" event={"ID":"7e6e32a5-a0a1-4c2f-9708-66361456e02d","Type":"ContainerDied","Data":"9f7884dadb0ca99b7af748418596b91ad897d4379e5d60f8646b652a055eb156"}
Nov 01 20:14:26.851249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.851224    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:26.856391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.856375    6479 scope.go:122] "RemoveContainer" containerID="f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620"
Nov 01 20:14:26.856696 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:26.856670    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620\": container with ID starting with f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620 not found: ID does not exist" containerID="f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620"
Nov 01 20:14:26.856811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.856702    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620"} err="failed to get container status \"f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620\": rpc error: code = NotFound desc = could not find container \"f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620\": container with ID starting with f372d909cfb06fd5a23750365d515be207cfe12ac809c77bd2936e7257f88620 not found: ID does not exist"
Nov 01 20:14:26.856811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.856720    6479 scope.go:122] "RemoveContainer" containerID="bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe"
Nov 01 20:14:26.867822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.867792    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"]
Nov 01 20:14:26.868733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.868609    6479 scope.go:122] "RemoveContainer" containerID="bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe"
Nov 01 20:14:26.868997 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:26.868971    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe\": container with ID starting with bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe not found: ID does not exist" containerID="bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe"
Nov 01 20:14:26.869087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.869005    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe"} err="failed to get container status \"bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe\": rpc error: code = NotFound desc = could not find container \"bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe\": container with ID starting with bbdfae9030b59ba26b2b9735107d422b2bbfe6782b56fa77c7a26e5ec1e0c2fe not found: ID does not exist"
Nov 01 20:14:26.871826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.871799    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f0c86-58fd75464d-fn4kx"]
Nov 01 20:14:26.896672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.882915    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"]
Nov 01 20:14:26.907800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.902106    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91f1523a-66bbc7cc77-ljljk"]
Nov 01 20:14:26.913669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.912780    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.913916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.913893    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84n2\" (UniqueName: \"kubernetes.io/projected/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-kube-api-access-g84n2\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:26.914031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:26.914017    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:27.338893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.338804    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:27.563432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.563395    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:27.750880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.750841    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"]
Nov 01 20:14:27.751498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751477    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerName="payload-processing"
Nov 01 20:14:27.751498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751497    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerName="payload-processing"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751521    6479 cpu_manager.go:414] "containerMap: removing container" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751529    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751558    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" containerName="payload-pre-processing"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751564    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" containerName="payload-pre-processing"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751582    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7e6e32a5-a0a1-4c2f-9708-66361456e02d" containerName="maas-api"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751587    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6e32a5-a0a1-4c2f-9708-66361456e02d" containerName="maas-api"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751594    6479 cpu_manager.go:414] "containerMap: removing container" podUID="2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" containerName="maas-api"
Nov 01 20:14:27.751666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751600    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" containerName="maas-api"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751772    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6e32a5-a0a1-4c2f-9708-66361456e02d" containerName="maas-api"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751792    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" containerName="maas-api"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751802    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee6f741-9e37-48bc-a647-0620e3cd2362" containerName="payload-processing"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751816    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751831    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ad8539-02cd-4b6e-8d7a-d8bf72644ab4" containerName="istio-proxy"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751841    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy"
Nov 01 20:14:27.752095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.751849    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8347908c-6d47-4167-9621-462e8f9c4d00" containerName="payload-pre-processing"
Nov 01 20:14:27.757662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.757628    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.766521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.766434    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-912c8c21-openshift-default-dockercfg-v8crr\""
Nov 01 20:14:27.774288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.772523    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"]
Nov 01 20:14:27.795698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.794602    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:27.817722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.817686    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:14:27.831240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831207    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831248    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831268    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fggx\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831451    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831545    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831572    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831608    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831717    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.831825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.831813    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.885184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.885143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943502    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943563    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943610    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943671    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943801    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943869    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943909    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.943932    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fggx\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.945007    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.952077    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.952771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.952730    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.972669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.955702    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.972669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.957789    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.972669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.958881    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.972669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.959123    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.972669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.959346    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.972669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.969935    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fggx\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:27.994667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:27.994357    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token\") pod \"e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") " pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:28.081722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.080784    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:28.390150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.390065    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:28.641710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.641614    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"]
Nov 01 20:14:28.659356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.658723    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:14:28.659356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.658801    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:14:28.659356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.658865    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:14:28.720879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.720824    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2bf228-14b5-4d15-b76f-2b1b2f3a0183" path="/var/lib/kubelet/pods/2f2bf228-14b5-4d15-b76f-2b1b2f3a0183/volumes"
Nov 01 20:14:28.722201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.722141    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6e32a5-a0a1-4c2f-9708-66361456e02d" path="/var/lib/kubelet/pods/7e6e32a5-a0a1-4c2f-9708-66361456e02d/volumes"
Nov 01 20:14:28.931530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:28.931496    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" event={"ID":"20f1a937-de00-4cdf-9f95-88311eb793e9","Type":"ContainerStarted","Data":"e6a6ea1b0186da3bc91229f1dba390705482631a39b133cf91bee6cc9d7a09d1"}
Nov 01 20:14:29.174401 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:29.174041    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:14:29.254566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.254524    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:29.279190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.279100    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls" podUID="f324287a-e272-46f4-8277-7e7b8ba6bf9e" containerName="payload-processing" containerID="cri-o://9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250" gracePeriod=25
Nov 01 20:14:29.291868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.291828    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9" podUID="ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" containerName="payload-pre-processing" containerID="cri-o://27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64" gracePeriod=25
Nov 01 20:14:29.594502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.594116    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:29.625560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.625532    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:29.722937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.722902    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz\" (UniqueName: \"kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz\") pod \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") "
Nov 01 20:14:29.723111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.722991    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume\") pod \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") "
Nov 01 20:14:29.723111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.723026    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume\") pod \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\" (UID: \"f324287a-e272-46f4-8277-7e7b8ba6bf9e\") "
Nov 01 20:14:29.723279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.723261    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd\" (UniqueName: \"kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd\") pod \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\" (UID: \"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5\") "
Nov 01 20:14:29.723469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.723448    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume" pod "ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" (UID: "ccec9bed-3733-4ea5-bbe3-5ef27cd690d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:29.723544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.723501    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume" pod "f324287a-e272-46f4-8277-7e7b8ba6bf9e" (UID: "f324287a-e272-46f4-8277-7e7b8ba6bf9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:29.723890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.723866    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f324287a-e272-46f4-8277-7e7b8ba6bf9e-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:29.723978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.723895    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:29.725200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.725175    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz" pod "f324287a-e272-46f4-8277-7e7b8ba6bf9e" (UID: "f324287a-e272-46f4-8277-7e7b8ba6bf9e"). InnerVolumeSpecName "kube-api-access-pptzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:29.725288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.725224    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd" pod "ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" (UID: "ccec9bed-3733-4ea5-bbe3-5ef27cd690d5"). InnerVolumeSpecName "kube-api-access-vw4xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:29.825231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.825148    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptzz\" (UniqueName: \"kubernetes.io/projected/f324287a-e272-46f4-8277-7e7b8ba6bf9e-kube-api-access-pptzz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:29.825231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.825181    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4xd\" (UniqueName: \"kubernetes.io/projected/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5-kube-api-access-vw4xd\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:29.895258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.895225    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:29.940573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.940524    6479 generic.go:356] "Generic (PLEG): container finished" podID="ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" containerID="27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64" exitCode=0
Nov 01 20:14:29.947797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.940608    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"
Nov 01 20:14:29.947797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.940723    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9" event={"ID":"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5","Type":"ContainerDied","Data":"27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64"}
Nov 01 20:14:29.947797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.940774    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9" event={"ID":"ccec9bed-3733-4ea5-bbe3-5ef27cd690d5","Type":"ContainerDied","Data":"eb8a00aab5fac614843622b6295a33a42a92656b60794c586a84dc7db89f4299"}
Nov 01 20:14:29.947797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:29.940802    6479 scope.go:122] "RemoveContainer" containerID="27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64"
Nov 01 20:14:30.030236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.030208    6479 generic.go:356] "Generic (PLEG): container finished" podID="f324287a-e272-46f4-8277-7e7b8ba6bf9e" containerID="9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250" exitCode=0
Nov 01 20:14:30.030441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.030428    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"
Nov 01 20:14:30.031125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.030991    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls" event={"ID":"f324287a-e272-46f4-8277-7e7b8ba6bf9e","Type":"ContainerDied","Data":"9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250"}
Nov 01 20:14:30.031125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.031045    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls" event={"ID":"f324287a-e272-46f4-8277-7e7b8ba6bf9e","Type":"ContainerDied","Data":"2c27ea4a78f6a78e7011fb171b60bb48ab2d2b48e74b28ecac1ccf30a1b55057"}
Nov 01 20:14:30.032946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.032401    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" event={"ID":"20f1a937-de00-4cdf-9f95-88311eb793e9","Type":"ContainerStarted","Data":"721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10"}
Nov 01 20:14:30.039852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.039765    6479 scope.go:122] "RemoveContainer" containerID="27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64"
Nov 01 20:14:30.040136 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:30.040111    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64\": container with ID starting with 27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64 not found: ID does not exist" containerID="27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64"
Nov 01 20:14:30.040195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.040149    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64"} err="failed to get container status \"27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64\": rpc error: code = NotFound desc = could not find container \"27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64\": container with ID starting with 27f945701d2d47a17db9727d9613298a42228bc7832570d8cab1fc8648131b64 not found: ID does not exist"
Nov 01 20:14:30.040195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.040172    6479 scope.go:122] "RemoveContainer" containerID="9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250"
Nov 01 20:14:30.049753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.049723    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"]
Nov 01 20:14:30.050704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.050681    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-91f1523a-7fcf94b5c5-4txh9"]
Nov 01 20:14:30.051184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.051167    6479 scope.go:122] "RemoveContainer" containerID="9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250"
Nov 01 20:14:30.051474 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:30.051456    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250\": container with ID starting with 9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250 not found: ID does not exist" containerID="9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250"
Nov 01 20:14:30.051541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.051484    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250"} err="failed to get container status \"9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250\": rpc error: code = NotFound desc = could not find container \"9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250\": container with ID starting with 9782f7b6a93752cf5e70300a00c9d3ec1f6e32150fc7ddf9d5a43b2d406f2250 not found: ID does not exist"
Nov 01 20:14:30.060255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.060231    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"]
Nov 01 20:14:30.063949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.063926    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-91f1523a-899695877-8wlls"]
Nov 01 20:14:30.079887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.079790    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" podStartSLOduration=3.079775762 podStartE2EDuration="3.079775762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:30.078521801 +0000 UTC m=+915.494154294" watchObservedRunningTime="2026-11-01 20:14:30.079775762 +0000 UTC m=+915.495408234"
Nov 01 20:14:30.081904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.081878    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:30.087592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.087568    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:30.709010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.708966    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" path="/var/lib/kubelet/pods/ccec9bed-3733-4ea5-bbe3-5ef27cd690d5/volumes"
Nov 01 20:14:30.709502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.709480    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f324287a-e272-46f4-8277-7e7b8ba6bf9e" path="/var/lib/kubelet/pods/f324287a-e272-46f4-8277-7e7b8ba6bf9e/volumes"
Nov 01 20:14:30.931068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:30.931017    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:31.066962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:31.066878    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:31.075994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:31.069131    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:14:31.511261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:31.511231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:31.603369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:31.603333    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:31.908386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:31.908299    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:32.540286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.540241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:32.698600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.698563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:32.740102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740067    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"]
Nov 01 20:14:32.740712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740693    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f324287a-e272-46f4-8277-7e7b8ba6bf9e" containerName="payload-processing"
Nov 01 20:14:32.740776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740713    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f324287a-e272-46f4-8277-7e7b8ba6bf9e" containerName="payload-processing"
Nov 01 20:14:32.740776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740724    6479 cpu_manager.go:414] "containerMap: removing container" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy"
Nov 01 20:14:32.740776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740729    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405d10b4-c73c-461e-a9ac-0c17bb03fc3f" containerName="istio-proxy"
Nov 01 20:14:32.740776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740773    6479 cpu_manager.go:414] "containerMap: removing container" podUID="ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" containerName="payload-pre-processing"
Nov 01 20:14:32.740919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740778    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" containerName="payload-pre-processing"
Nov 01 20:14:32.740919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740855    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccec9bed-3733-4ea5-bbe3-5ef27cd690d5" containerName="payload-pre-processing"
Nov 01 20:14:32.740919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.740865    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f324287a-e272-46f4-8277-7e7b8ba6bf9e" containerName="payload-processing"
Nov 01 20:14:32.745854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.745838    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:32.756734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.756270    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"]
Nov 01 20:14:32.756734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.756370    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-912c8c21\""
Nov 01 20:14:32.793010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.792937    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:32.817059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.817031    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"]
Nov 01 20:14:32.821055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.821038    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:32.829450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.829257    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-912c8c21-dockercfg-htj4m\""
Nov 01 20:14:32.832680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.832622    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"]
Nov 01 20:14:32.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.917971    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvlr\" (UniqueName: \"kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr\") pod \"payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:32.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.918054    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume\") pod \"payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") " pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:32.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.918144    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:32.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:32.918333    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s4m\" (UniqueName: \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\") pod \"payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") " pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:33.053724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.053194    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:33.053724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.053295    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s4m\" (UniqueName: \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\") pod \"payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") " pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:33.053724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.053468    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvlr\" (UniqueName: \"kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr\") pod \"payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:33.053724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.053525    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume\") pod \"payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") " pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:33.061925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.054463    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume\") pod \"payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") " pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:33.061925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.055459    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:33.097666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.096718    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvlr\" (UniqueName: \"kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr\") pod \"payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:33.101720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.099196    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s4m\" (UniqueName: \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\") pod \"payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") " pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:33.129975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.129871    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:33.136721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.136692    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:33.269234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.269199    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:33.379546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.379463    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:33.393640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.393598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:33.425049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.424958    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"]
Nov 01 20:14:33.576111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.573404    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:33.715966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.715932    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"]
Nov 01 20:14:33.718155 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:33.718120    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51d88d1_78c5_48b3_931d_f8d8e3b5b69a.slice/crio-2ef8f0b3ef48ac62cbbfbf119a9fb70d856492fc6acc74d2b83763537c7364ae WatchSource:0}: Error finding container 2ef8f0b3ef48ac62cbbfbf119a9fb70d856492fc6acc74d2b83763537c7364ae: Status 404 returned error can't find the container with id 2ef8f0b3ef48ac62cbbfbf119a9fb70d856492fc6acc74d2b83763537c7364ae
Nov 01 20:14:33.881059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.881025    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"]
Nov 01 20:14:33.881256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.881235    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" containerID="cri-o://fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674" gracePeriod=30
Nov 01 20:14:33.908691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:33.908638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:34.210683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.205303    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz" event={"ID":"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a","Type":"ContainerStarted","Data":"c7d806b966f27e7c805ce32c343f08ecd06417eba68824eb644ef66c244bf5f5"}
Nov 01 20:14:34.210683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.205367    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz" event={"ID":"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a","Type":"ContainerStarted","Data":"2ef8f0b3ef48ac62cbbfbf119a9fb70d856492fc6acc74d2b83763537c7364ae"}
Nov 01 20:14:34.210683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.206356    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:34.210683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.208966    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576" event={"ID":"2bf9914b-57ed-4429-b52d-3c2123ad215c","Type":"ContainerStarted","Data":"1dd1621442ba3d1e47a14cb01ea1d264d2c9a64eb6e0efbc904e3a8e54ca32f1"}
Nov 01 20:14:34.210683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.209009    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576" event={"ID":"2bf9914b-57ed-4429-b52d-3c2123ad215c","Type":"ContainerStarted","Data":"785889d0e577a6e6148b8696a327d49fccb8d56e1f9a51249166b51912b3b3e9"}
Nov 01 20:14:34.210683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.209623    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:34.227772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.227727    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz" podStartSLOduration=2.227712206 podStartE2EDuration="2.227712206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:34.225485085 +0000 UTC m=+919.641117561" watchObservedRunningTime="2026-11-01 20:14:34.227712206 +0000 UTC m=+919.643344679"
Nov 01 20:14:34.244240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.244181    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576" podStartSLOduration=2.244164098 podStartE2EDuration="2.244164098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:34.241492956 +0000 UTC m=+919.657125431" watchObservedRunningTime="2026-11-01 20:14:34.244164098 +0000 UTC m=+919.659796572"
Nov 01 20:14:34.254199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.254165    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:34.871045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:34.871002    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:35.055005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.054923    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:35.366270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.366227    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:35.558639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.558606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:35.604483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.604449    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"]
Nov 01 20:14:35.608865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.608846    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.618017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.617994    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 20:14:35.618123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.618082    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 20:14:35.618539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.618514    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"]
Nov 01 20:14:35.619192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.619175    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 20:14:35.619294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.619227    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 20:14:35.702839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.702807    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cdf911f2593b\""
Nov 01 20:14:35.703004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.702892    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-f429g\""
Nov 01 20:14:35.777530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.777490    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.777827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.777590    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.777827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.777827    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knft5\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.778039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.777911    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.838751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.838713    6479 patch_prober.go:29] interesting pod/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:14:35.839875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.839840    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:14:35.882547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.882456    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.884166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.884137    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knft5\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.884769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.884746    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.885464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.885443    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.908961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.883923    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.909177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.886282    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.909283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.901238    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.911220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.911189    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knft5\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5\") pod \"payload-processing-external-model-566dcd79c7-n9nvf\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") " pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.925563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.925295    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:35.985670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:35.978311    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:36.346631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:36.346607    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"]
Nov 01 20:14:36.348819 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:36.348791    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde615f26_cdfa_48b5_acb4_5d6765ca3e6a.slice/crio-404a09c9b6200d3e5040bfb363982413b1fdbca9d9cd72aeade4591517dadb15 WatchSource:0}: Error finding container 404a09c9b6200d3e5040bfb363982413b1fdbca9d9cd72aeade4591517dadb15: Status 404 returned error can't find the container with id 404a09c9b6200d3e5040bfb363982413b1fdbca9d9cd72aeade4591517dadb15
Nov 01 20:14:36.416475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:36.416441    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf" event={"ID":"de615f26-cdfa-48b5-acb4-5d6765ca3e6a","Type":"ContainerStarted","Data":"404a09c9b6200d3e5040bfb363982413b1fdbca9d9cd72aeade4591517dadb15"}
Nov 01 20:14:36.757511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:36.757480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:37.412533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.412497    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"]
Nov 01 20:14:37.421281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.417236    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.423963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.423934    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-912c8c21-serving-cert\""
Nov 01 20:14:37.431267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.431222    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf" event={"ID":"de615f26-cdfa-48b5-acb4-5d6765ca3e6a","Type":"ContainerStarted","Data":"faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce"}
Nov 01 20:14:37.431565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.431540    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"]
Nov 01 20:14:37.432290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.432264    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:37.454970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.454942    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r268n\" (UniqueName: \"kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.455126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.455088    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.455473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.455448    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.455679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.455613    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.460414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.460366    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf" podStartSLOduration=2.460350495 podStartE2EDuration="2.460350495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:37.457847546 +0000 UTC m=+922.873480022" watchObservedRunningTime="2026-11-01 20:14:37.460350495 +0000 UTC m=+922.875982969"
Nov 01 20:14:37.465908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.465886    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:37.556689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.556626    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.556870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.556775    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.556870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.556806    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r268n\" (UniqueName: \"kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.556870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.556850    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.557363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.557334    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.559481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.559448    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.559752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.559734    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.563127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.563097    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:37.564720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.564693    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r268n\" (UniqueName: \"kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n\") pod \"maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:37.743144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:37.743108    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:38.084315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:38.084274    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"]
Nov 01 20:14:38.148669 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:38.137032    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55074d28_db16_4ebb_a3ab_4cd401116fd3.slice/crio-0027f03df8c10ec97a2ba5583caf2dbb1faaf8612a2200672cf44e96d455c521 WatchSource:0}: Error finding container 0027f03df8c10ec97a2ba5583caf2dbb1faaf8612a2200672cf44e96d455c521: Status 404 returned error can't find the container with id 0027f03df8c10ec97a2ba5583caf2dbb1faaf8612a2200672cf44e96d455c521
Nov 01 20:14:38.188878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:38.188801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:38.438863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:38.438778    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv" event={"ID":"55074d28-db16-4ebb-a3ab-4cd401116fd3","Type":"ContainerStarted","Data":"0027f03df8c10ec97a2ba5583caf2dbb1faaf8612a2200672cf44e96d455c521"}
Nov 01 20:14:38.860954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:38.860914    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:39.179416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.179379    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:14:39.188100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188062    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188106    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188150    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188237    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188379    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188430    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188466    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188475    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:39.188638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188520    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:39.188638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188536    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188568    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:39.188638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188594    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd\") pod \"5b3ab07f-1da6-442f-857b-5789828beea8\" (UID: \"5b3ab07f-1da6-442f-857b-5789828beea8\") "
Nov 01 20:14:39.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188840    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:39.188867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.188849    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:39.189265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.189243    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.189326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.189270    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.189326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.189286    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b3ab07f-1da6-442f-857b-5789828beea8-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.189326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.189299    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.189326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.189313    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.190501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.190469    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:14:39.190945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.190907    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:39.191089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.191071    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:39.191279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.191257    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd" pod "5b3ab07f-1da6-442f-857b-5789828beea8" (UID: "5b3ab07f-1da6-442f-857b-5789828beea8"). InnerVolumeSpecName "kube-api-access-phtcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:39.290509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.290435    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b3ab07f-1da6-442f-857b-5789828beea8-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.290509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.290470    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.290509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.290484    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtcd\" (UniqueName: \"kubernetes.io/projected/5b3ab07f-1da6-442f-857b-5789828beea8-kube-api-access-phtcd\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.290509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.290494    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b3ab07f-1da6-442f-857b-5789828beea8-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:39.446477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.446444    6479 generic.go:356] "Generic (PLEG): container finished" podID="5b3ab07f-1da6-442f-857b-5789828beea8" containerID="fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674" exitCode=0
Nov 01 20:14:39.446918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.446487    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"
Nov 01 20:14:39.446918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.446530    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" event={"ID":"5b3ab07f-1da6-442f-857b-5789828beea8","Type":"ContainerDied","Data":"fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674"}
Nov 01 20:14:39.446918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.446619    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc" event={"ID":"5b3ab07f-1da6-442f-857b-5789828beea8","Type":"ContainerDied","Data":"954e9d8af83f516991f0ed61b7a125534c43eabc3ae8ce5f20b4c107cf9771a8"}
Nov 01 20:14:39.446918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.446688    6479 scope.go:122] "RemoveContainer" containerID="fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674"
Nov 01 20:14:39.448476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.448454    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv" event={"ID":"55074d28-db16-4ebb-a3ab-4cd401116fd3","Type":"ContainerStarted","Data":"36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17"}
Nov 01 20:14:39.448586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.448508    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:39.461954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.461937    6479 scope.go:122] "RemoveContainer" containerID="10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"
Nov 01 20:14:39.468274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.468230    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv" podStartSLOduration=1.949144586 podStartE2EDuration="2.468213346s" totalImagesPullingTime="519.06876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:39.466876305 +0000 UTC m=+924.882508779" watchObservedRunningTime="2026-11-01 20:14:39.468213346 +0000 UTC m=+924.883845820"
Nov 01 20:14:39.473767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.473747    6479 scope.go:122] "RemoveContainer" containerID="fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674"
Nov 01 20:14:39.474086 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:39.474064    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674\": container with ID starting with fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674 not found: ID does not exist" containerID="fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674"
Nov 01 20:14:39.474146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.474099    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674"} err="failed to get container status \"fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674\": rpc error: code = NotFound desc = could not find container \"fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674\": container with ID starting with fcc25d4d93b35f340914f53d7e97164bab4ca42abc05edc2b0dab566b0039674 not found: ID does not exist"
Nov 01 20:14:39.474146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.474122    6479 scope.go:122] "RemoveContainer" containerID="10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"
Nov 01 20:14:39.474429 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:39.474410    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15\": container with ID starting with 10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15 not found: ID does not exist" containerID="10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"
Nov 01 20:14:39.474465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.474437    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15"} err="failed to get container status \"10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15\": rpc error: code = NotFound desc = could not find container \"10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15\": container with ID starting with 10e3bb21fcca28bb356d0e1b64ce74294a92c9f26f71360c0e183ee496415d15 not found: ID does not exist"
Nov 01 20:14:39.484532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.484507    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"]
Nov 01 20:14:39.488145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.488123    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f0c86-openshift-default-6987bb768f-n25xc"]
Nov 01 20:14:39.493928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:39.493902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:40.145387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:40.145355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:40.708601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:40.708568    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" path="/var/lib/kubelet/pods/5b3ab07f-1da6-442f-857b-5789828beea8/volumes"
Nov 01 20:14:40.745153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:40.745124    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:41.376750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:41.376699    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:41.476175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:41.476138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:42.060921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:42.060887    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:42.708561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:42.708524    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:43.375320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:43.375283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:43.475806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:43.475770    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:44.134168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:44.134135    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:44.737911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:44.737880    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:45.332036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:45.332004    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:45.497175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:45.497140    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:14:45.945950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:45.945915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:46.419703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:46.419612    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:14:46.419851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:46.419715    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:14:46.565116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:46.565080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:47.152393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:47.152339    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:47.868313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:47.868276    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:48.441413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.441382    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:14:48.749082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.749050    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:48.911717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.904240    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"]
Nov 01 20:14:48.913041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.913009    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy"
Nov 01 20:14:48.972736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.972699    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy"
Nov 01 20:14:48.972996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.972979    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy"
Nov 01 20:14:48.973096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.973083    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy"
Nov 01 20:14:48.998671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.973448    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy"
Nov 01 20:14:48.998977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:48.998934    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3ab07f-1da6-442f-857b-5789828beea8" containerName="istio-proxy"
Nov 01 20:14:49.008983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.007119    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"]
Nov 01 20:14:49.008983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.007254    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.037609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.037533    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:49.086940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.086901    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.087177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.087152    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.087274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.087259    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.087317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.087289    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82dkz\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.194669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.191662    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.194669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.191718    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82dkz\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.194669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.191818    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.194669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.192043    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.194669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.193065    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.194669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.193687    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.209344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.206876    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.217666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.214172    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82dkz\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.277850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.277764    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:49.363671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.355950    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:49.635867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.635792    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:49.780666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:49.765988    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"]
Nov 01 20:14:49.788720 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:14:49.781328    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c2fad9_a87d_47ca_b6b9_2df730d1d529.slice/crio-85a8ddd416789df64ea6f99114c27d64f46aa42886e05ba0db4efc93568d9b08 WatchSource:0}: Error finding container 85a8ddd416789df64ea6f99114c27d64f46aa42886e05ba0db4efc93568d9b08: Status 404 returned error can't find the container with id 85a8ddd416789df64ea6f99114c27d64f46aa42886e05ba0db4efc93568d9b08
Nov 01 20:14:50.036668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:50.036619    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:50.535535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:50.535491    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg" event={"ID":"01c2fad9-a87d-47ca-b6b9-2df730d1d529","Type":"ContainerStarted","Data":"b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102"}
Nov 01 20:14:50.535761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:50.535546    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg" event={"ID":"01c2fad9-a87d-47ca-b6b9-2df730d1d529","Type":"ContainerStarted","Data":"85a8ddd416789df64ea6f99114c27d64f46aa42886e05ba0db4efc93568d9b08"}
Nov 01 20:14:50.535761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:50.535628    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:14:50.551863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:50.551814    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg" podStartSLOduration=2.551799808 podStartE2EDuration="2.551799808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:50.550431529 +0000 UTC m=+935.966064049" watchObservedRunningTime="2026-11-01 20:14:50.551799808 +0000 UTC m=+935.967432271"
Nov 01 20:14:50.831008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:50.830904    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:51.561597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:51.561552    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:52.167983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:52.167918    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:52.397533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:52.397484    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:14:53.161372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:53.161338    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:54.187513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:54.187464    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:55.048768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:55.048707    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:55.147226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:55.147193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:55.505817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:55.505776    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:56.642228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:56.642195    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:57.463050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:57.462995    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:58.161676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:58.161622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:58.765513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:58.765469    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:59.171816 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:14:59.171729    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:14:59.340449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:59.340414    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:14:59.954025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:14:59.953992    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:00.168618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.168574    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"]
Nov 01 20:15:00.182456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.182421    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"]
Nov 01 20:15:00.192810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.188099    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.206206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.206124    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"]
Nov 01 20:15:00.206206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.206187    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"]
Nov 01 20:15:00.206430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.206287    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.224280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.223455    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv269\" (UniqueName: \"kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269\") pod \"maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.224280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.223593    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.283637 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.283598    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:15:00.325182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.325146    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.325382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.325188    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv269\" (UniqueName: \"kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269\") pod \"maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.325382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.325266    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhv5k\" (UniqueName: \"kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.325382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.325315    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.325986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.325967    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.332989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.332964    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv269\" (UniqueName: \"kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269\") pod \"maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.426563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.426529    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.426744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.426589    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhv5k\" (UniqueName: \"kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.427166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.427145    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.434998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.434963    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhv5k\" (UniqueName: \"kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.532269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.532183    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:00.544695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.544636    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:00.635021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.634958    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:00.829211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.829052    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"]
Nov 01 20:15:00.845806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:00.845781    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"]
Nov 01 20:15:00.846778 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:00.846744    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f60de4c_c812_4162_8b95_c707b44290a3.slice/crio-b9a57cae6538179a710aadf9dc5a487e0c3fd136ffd30254a80ec3bae8996584 WatchSource:0}: Error finding container b9a57cae6538179a710aadf9dc5a487e0c3fd136ffd30254a80ec3bae8996584: Status 404 returned error can't find the container with id b9a57cae6538179a710aadf9dc5a487e0c3fd136ffd30254a80ec3bae8996584
Nov 01 20:15:01.351567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:01.351534    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:01.546999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:01.546973    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:15:01.596529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:01.596479    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"]
Nov 01 20:15:01.828610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:01.828572    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerStarted","Data":"b9a57cae6538179a710aadf9dc5a487e0c3fd136ffd30254a80ec3bae8996584"}
Nov 01 20:15:01.829838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:01.829810    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj" event={"ID":"7114b1d0-ae7f-41b5-b099-044b2cbae49c","Type":"ContainerStarted","Data":"e41cb9b029e7456402de422e7de9fa06e5fa632d9a9c6f3fc6c8f9c6d1be1ac3"}
Nov 01 20:15:02.059955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:02.059915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:02.770925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:02.770865    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:02.799719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:02.799681    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"]
Nov 01 20:15:02.819910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:02.819470    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"]
Nov 01 20:15:02.871739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:02.871617    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:02.971870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:02.971833    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:03.548484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:03.548358    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:03.821495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:03.821372    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:04.031974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.031935    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"]
Nov 01 20:15:04.033308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.033272    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv" podUID="55074d28-db16-4ebb-a3ab-4cd401116fd3" containerName="maas-api" containerID="cri-o://36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17" gracePeriod=30
Nov 01 20:15:04.089916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.089814    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"]
Nov 01 20:15:04.093495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.092269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:04.252677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.251219    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:04.369720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.352343    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:04.447986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.447696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:04.539893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.539819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:04.839166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.839140    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:15:04.904168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.904088    6479 generic.go:356] "Generic (PLEG): container finished" podID="55074d28-db16-4ebb-a3ab-4cd401116fd3" containerID="36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17" exitCode=0
Nov 01 20:15:04.904168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.904144    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv" event={"ID":"55074d28-db16-4ebb-a3ab-4cd401116fd3","Type":"ContainerDied","Data":"36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17"}
Nov 01 20:15:04.904392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.904186    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv" event={"ID":"55074d28-db16-4ebb-a3ab-4cd401116fd3","Type":"ContainerDied","Data":"0027f03df8c10ec97a2ba5583caf2dbb1faaf8612a2200672cf44e96d455c521"}
Nov 01 20:15:04.904392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.904209    6479 scope.go:122] "RemoveContainer" containerID="36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17"
Nov 01 20:15:04.904505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.904393    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"
Nov 01 20:15:04.920576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.920550    6479 scope.go:122] "RemoveContainer" containerID="36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17"
Nov 01 20:15:04.921324 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:04.921246    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17\": container with ID starting with 36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17 not found: ID does not exist" containerID="36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17"
Nov 01 20:15:04.921324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.921285    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17"} err="failed to get container status \"36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17\": rpc error: code = NotFound desc = could not find container \"36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17\": container with ID starting with 36d8c13f17abb995c6ac3b1c2288aedf9ea88eb49367736af800d6a1daaf0f17 not found: ID does not exist"
Nov 01 20:15:04.943247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.943207    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n\" (UniqueName: \"kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n\") pod \"55074d28-db16-4ebb-a3ab-4cd401116fd3\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") "
Nov 01 20:15:04.943423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.943275    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs\") pod \"55074d28-db16-4ebb-a3ab-4cd401116fd3\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") "
Nov 01 20:15:04.943423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.943409    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca\") pod \"55074d28-db16-4ebb-a3ab-4cd401116fd3\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") "
Nov 01 20:15:04.943535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.943452    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls\") pod \"55074d28-db16-4ebb-a3ab-4cd401116fd3\" (UID: \"55074d28-db16-4ebb-a3ab-4cd401116fd3\") "
Nov 01 20:15:04.944153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.944124    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca" pod "55074d28-db16-4ebb-a3ab-4cd401116fd3" (UID: "55074d28-db16-4ebb-a3ab-4cd401116fd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:04.946561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.946533    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls" pod "55074d28-db16-4ebb-a3ab-4cd401116fd3" (UID: "55074d28-db16-4ebb-a3ab-4cd401116fd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:04.947135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.947092    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n" pod "55074d28-db16-4ebb-a3ab-4cd401116fd3" (UID: "55074d28-db16-4ebb-a3ab-4cd401116fd3"). InnerVolumeSpecName "kube-api-access-r268n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:04.947135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:04.947118    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs" pod "55074d28-db16-4ebb-a3ab-4cd401116fd3" (UID: "55074d28-db16-4ebb-a3ab-4cd401116fd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:05.045447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.045405    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r268n\" (UniqueName: \"kubernetes.io/projected/55074d28-db16-4ebb-a3ab-4cd401116fd3-kube-api-access-r268n\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:05.045447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.045450    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:05.045754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.045467    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55074d28-db16-4ebb-a3ab-4cd401116fd3-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:05.045754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.045482    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55074d28-db16-4ebb-a3ab-4cd401116fd3-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:05.232116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.232082    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"]
Nov 01 20:15:05.235611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.235580    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-912c8c21-699dd56c8f-fjpgv"]
Nov 01 20:15:05.387695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.376425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:05.918023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.917978    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"]
Nov 01 20:15:05.918395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.918230    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" podUID="20f1a937-de00-4cdf-9f95-88311eb793e9" containerName="istio-proxy" containerID="cri-o://721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10" gracePeriod=30
Nov 01 20:15:05.922723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:05.921893    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:06.206227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.198530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:06.488032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.487994    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:06.573665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.573447    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"]
Nov 01 20:15:06.591632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.574512    6479 cpu_manager.go:414] "containerMap: removing container" podUID="55074d28-db16-4ebb-a3ab-4cd401116fd3" containerName="maas-api"
Nov 01 20:15:06.591632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.574536    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55074d28-db16-4ebb-a3ab-4cd401116fd3" containerName="maas-api"
Nov 01 20:15:06.591632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.574725    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="55074d28-db16-4ebb-a3ab-4cd401116fd3" containerName="maas-api"
Nov 01 20:15:06.607781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.597661    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv" podUID="d2473191-76d3-4ca9-a191-cb2fa7f46be1" containerName="payload-processing" containerID="cri-o://d4377507f6b8e7db4302bd2202659bfca0bdac3cd114a2754ef1192d87f9ddd4" gracePeriod=25
Nov 01 20:15:06.607781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.599936    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"]
Nov 01 20:15:06.607781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.600233    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.614351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.613780    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:15:06.640953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.640924    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.641139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.641112    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sr6w\" (UniqueName: \"kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.641250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.641234    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.710594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.710564    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55074d28-db16-4ebb-a3ab-4cd401116fd3" path="/var/lib/kubelet/pods/55074d28-db16-4ebb-a3ab-4cd401116fd3/volumes"
Nov 01 20:15:06.742331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.742296    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.742520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.742501    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sr6w\" (UniqueName: \"kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.742927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.742906    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.743587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.743557    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.745818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.745795    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.779681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.775786    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sr6w\" (UniqueName: \"kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w\") pod \"maas-api-revoke-sub-ddffb270-26h7d\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.921716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.921260    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"]
Nov 01 20:15:06.927816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.927794    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"]
Nov 01 20:15:06.927941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.927904    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:06.931011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.930827    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:06.935934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.935878    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9b54c8e3-openshift-default-dockercfg-785df\""
Nov 01 20:15:06.953343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.953315    6479 generic.go:356] "Generic (PLEG): container finished" podID="d2473191-76d3-4ca9-a191-cb2fa7f46be1" containerID="d4377507f6b8e7db4302bd2202659bfca0bdac3cd114a2754ef1192d87f9ddd4" exitCode=0
Nov 01 20:15:06.953626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.953601    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv" event={"ID":"d2473191-76d3-4ca9-a191-cb2fa7f46be1","Type":"ContainerDied","Data":"d4377507f6b8e7db4302bd2202659bfca0bdac3cd114a2754ef1192d87f9ddd4"}
Nov 01 20:15:06.955733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.955666    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:06.958966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:06.958118    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerStarted","Data":"705fb24cff455f6d1b2078864b07dbe63af24176eafe44eaddee742fe3d7b628"}
Nov 01 20:15:07.002590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.002532    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" podStartSLOduration=1.582596239 podStartE2EDuration="7.0025138s" totalImagesPullingTime="5.419917561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:06.981832388 +0000 UTC m=+952.397464863" watchObservedRunningTime="2026-11-01 20:15:07.0025138 +0000 UTC m=+952.418146275"
Nov 01 20:15:07.004472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.004109    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:15:07.050988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.050865    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:07.060965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.058033    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials\") pod \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") "
Nov 01 20:15:07.060965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.058137    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr\") pod \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") "
Nov 01 20:15:07.060965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.058270    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume\") pod \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") "
Nov 01 20:15:07.060965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.058345    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume\") pod \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\" (UID: \"d2473191-76d3-4ca9-a191-cb2fa7f46be1\") "
Nov 01 20:15:07.060965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.060791    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume" pod "d2473191-76d3-4ca9-a191-cb2fa7f46be1" (UID: "d2473191-76d3-4ca9-a191-cb2fa7f46be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:07.060965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.060938    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume" pod "d2473191-76d3-4ca9-a191-cb2fa7f46be1" (UID: "d2473191-76d3-4ca9-a191-cb2fa7f46be1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.058633    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061421    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061463    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpq4\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061595    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061666    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061695    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061746    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061791    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061844    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061979    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:07.062624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.061999    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2473191-76d3-4ca9-a191-cb2fa7f46be1-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:07.092174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.092127    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials" pod "d2473191-76d3-4ca9-a191-cb2fa7f46be1" (UID: "d2473191-76d3-4ca9-a191-cb2fa7f46be1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:07.106674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.106088    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr" pod "d2473191-76d3-4ca9-a191-cb2fa7f46be1" (UID: "d2473191-76d3-4ca9-a191-cb2fa7f46be1"). InnerVolumeSpecName "kube-api-access-gnwcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:07.147262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.147228    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184322    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184388    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpq4\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184485    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184534    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184564    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184608    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184669    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184720    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184847    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184924    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.184944    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwcr\" (UniqueName: \"kubernetes.io/projected/d2473191-76d3-4ca9-a191-cb2fa7f46be1-kube-api-access-gnwcr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.185370    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.188990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.185677    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.192583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.190773    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.192583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.191413    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.192583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.191711    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.192583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.191995    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.275787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.260300    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.334671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.323456    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.393697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.377948    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpq4\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4\") pod \"e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") " pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.504531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.504502    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"]
Nov 01 20:15:07.546157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.546131    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:07.801134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.800860    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576" podUID="2bf9914b-57ed-4429-b52d-3c2123ad215c" containerName="payload-processing" containerID="cri-o://1dd1621442ba3d1e47a14cb01ea1d264d2c9a64eb6e0efbc904e3a8e54ca32f1" gracePeriod=25
Nov 01 20:15:07.820491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.820444    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz" podUID="a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" containerName="payload-pre-processing" containerID="cri-o://c7d806b966f27e7c805ce32c343f08ecd06417eba68824eb644ef66c244bf5f5" gracePeriod=25
Nov 01 20:15:07.892022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.889834    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"]
Nov 01 20:15:07.926064 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:07.926028    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc54af2_3c7a_45f9_99a1_90ed1c03bd82.slice/crio-e171f466792bcfc0ec1ac25937f632bb33f521a1a21ffc976990b17df48e290c WatchSource:0}: Error finding container e171f466792bcfc0ec1ac25937f632bb33f521a1a21ffc976990b17df48e290c: Status 404 returned error can't find the container with id e171f466792bcfc0ec1ac25937f632bb33f521a1a21ffc976990b17df48e290c
Nov 01 20:15:07.928819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.928787    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:07.928929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.928873    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:07.928993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.928926    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:07.999766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.996156    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d" event={"ID":"ee24b5b6-f3d9-4154-8e64-24eaf58f658b","Type":"ContainerStarted","Data":"486960130c7f228c96514ac98f31716f9d5c6f789fa231feeb1b7c4815e57f82"}
Nov 01 20:15:07.999766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.998149    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj" event={"ID":"7114b1d0-ae7f-41b5-b099-044b2cbae49c","Type":"ContainerStarted","Data":"e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd"}
Nov 01 20:15:07.999766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:07.998327    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj" podUID="7114b1d0-ae7f-41b5-b099-044b2cbae49c" containerName="cleanup" containerID="cri-o://e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd" gracePeriod=30
Nov 01 20:15:08.001155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.001117    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" event={"ID":"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82","Type":"ContainerStarted","Data":"e171f466792bcfc0ec1ac25937f632bb33f521a1a21ffc976990b17df48e290c"}
Nov 01 20:15:08.010824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.009028    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv" event={"ID":"d2473191-76d3-4ca9-a191-cb2fa7f46be1","Type":"ContainerDied","Data":"e4ad9a02062ef1353e5e7b167f546974a868aeae6e58610539a2e827c8da199f"}
Nov 01 20:15:08.010824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.009088    6479 scope.go:122] "RemoveContainer" containerID="d4377507f6b8e7db4302bd2202659bfca0bdac3cd114a2754ef1192d87f9ddd4"
Nov 01 20:15:08.010824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.009227    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"
Nov 01 20:15:08.017131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.016210    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj" podStartSLOduration=2.105288001 podStartE2EDuration="8.016192511s" totalImagesPullingTime="5.91090451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:08.012351107 +0000 UTC m=+953.427983583" watchObservedRunningTime="2026-11-01 20:15:08.016192511 +0000 UTC m=+953.431824986"
Nov 01 20:15:08.017732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.017614    6479 generic.go:356] "Generic (PLEG): container finished" podID="a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" containerID="c7d806b966f27e7c805ce32c343f08ecd06417eba68824eb644ef66c244bf5f5" exitCode=0
Nov 01 20:15:08.017732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.017662    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz" event={"ID":"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a","Type":"ContainerDied","Data":"c7d806b966f27e7c805ce32c343f08ecd06417eba68824eb644ef66c244bf5f5"}
Nov 01 20:15:08.023312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.021792    6479 generic.go:356] "Generic (PLEG): container finished" podID="7f60de4c-c812-4162-8b95-c707b44290a3" containerID="705fb24cff455f6d1b2078864b07dbe63af24176eafe44eaddee742fe3d7b628" exitCode=60
Nov 01 20:15:08.023312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.021878    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerDied","Data":"705fb24cff455f6d1b2078864b07dbe63af24176eafe44eaddee742fe3d7b628"}
Nov 01 20:15:08.023312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.022842    6479 scope.go:122] "RemoveContainer" containerID="705fb24cff455f6d1b2078864b07dbe63af24176eafe44eaddee742fe3d7b628"
Nov 01 20:15:08.024785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.024725    6479 generic.go:356] "Generic (PLEG): container finished" podID="2bf9914b-57ed-4429-b52d-3c2123ad215c" containerID="1dd1621442ba3d1e47a14cb01ea1d264d2c9a64eb6e0efbc904e3a8e54ca32f1" exitCode=0
Nov 01 20:15:08.024785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.024759    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576" event={"ID":"2bf9914b-57ed-4429-b52d-3c2123ad215c","Type":"ContainerDied","Data":"1dd1621442ba3d1e47a14cb01ea1d264d2c9a64eb6e0efbc904e3a8e54ca32f1"}
Nov 01 20:15:08.111190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.111098    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"]
Nov 01 20:15:08.111641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.111475    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:08.116984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.116946    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-55dblfbcv"]
Nov 01 20:15:08.343617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.343313    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:15:08.468032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.452055    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume\") pod \"2bf9914b-57ed-4429-b52d-3c2123ad215c\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") "
Nov 01 20:15:08.468032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.452214    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\" (UniqueName: \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\") pod \"2bf9914b-57ed-4429-b52d-3c2123ad215c\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") "
Nov 01 20:15:08.499701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.498928    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume" pod "2bf9914b-57ed-4429-b52d-3c2123ad215c" (UID: "2bf9914b-57ed-4429-b52d-3c2123ad215c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:08.558313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.558252    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m" pod "2bf9914b-57ed-4429-b52d-3c2123ad215c" (UID: "2bf9914b-57ed-4429-b52d-3c2123ad215c"). InnerVolumeSpecName "kube-api-access-88s4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:08.558495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.558428    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\" (UniqueName: \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\") pod \"2bf9914b-57ed-4429-b52d-3c2123ad215c\" (UID: \"2bf9914b-57ed-4429-b52d-3c2123ad215c\") "
Nov 01 20:15:08.563670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.559047    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf9914b-57ed-4429-b52d-3c2123ad215c-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:08.563670 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:08.559147    6479 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/2bf9914b-57ed-4429-b52d-3c2123ad215c/volumes/kubernetes.io~projected/kube-api-access-88s4m
Nov 01 20:15:08.563670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.559194    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m" pod "2bf9914b-57ed-4429-b52d-3c2123ad215c" (UID: "2bf9914b-57ed-4429-b52d-3c2123ad215c"). InnerVolumeSpecName "kube-api-access-88s4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:08.640522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.640499    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:15:08.710669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.706988    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr\" (UniqueName: \"kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr\") pod \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") "
Nov 01 20:15:08.710669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.707193    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume\") pod \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\" (UID: \"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a\") "
Nov 01 20:15:08.710669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.707752    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s4m\" (UniqueName: \"kubernetes.io/projected/2bf9914b-57ed-4429-b52d-3c2123ad215c-kube-api-access-88s4m\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:08.710669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.708134    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume" pod "a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" (UID: "a51d88d1-78c5-48b3-931d-f8d8e3b5b69a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:08.733450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.729119    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2473191-76d3-4ca9-a191-cb2fa7f46be1" path="/var/lib/kubelet/pods/d2473191-76d3-4ca9-a191-cb2fa7f46be1/volumes"
Nov 01 20:15:08.733766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.733706    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr" pod "a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" (UID: "a51d88d1-78c5-48b3-931d-f8d8e3b5b69a"). InnerVolumeSpecName "kube-api-access-ggvlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:08.808571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.808542    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:08.808571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.808567    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvlr\" (UniqueName: \"kubernetes.io/projected/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a-kube-api-access-ggvlr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:08.810145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.810118    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:08.882872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.880513    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:08.929462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.921776    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269\" (UniqueName: \"kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269\") pod \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") "
Nov 01 20:15:08.929462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.922047    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca\") pod \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\" (UID: \"7114b1d0-ae7f-41b5-b099-044b2cbae49c\") "
Nov 01 20:15:08.968870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:08.951267    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca" pod "7114b1d0-ae7f-41b5-b099-044b2cbae49c" (UID: "7114b1d0-ae7f-41b5-b099-044b2cbae49c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:09.036446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.036390    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114b1d0-ae7f-41b5-b099-044b2cbae49c-maas-api-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:09.040851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.040792    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269" pod "7114b1d0-ae7f-41b5-b099-044b2cbae49c" (UID: "7114b1d0-ae7f-41b5-b099-044b2cbae49c"). InnerVolumeSpecName "kube-api-access-jv269". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:09.044971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.044941    6479 generic.go:356] "Generic (PLEG): container finished" podID="7114b1d0-ae7f-41b5-b099-044b2cbae49c" containerID="e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd" exitCode=6
Nov 01 20:15:09.045110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.045013    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj" event={"ID":"7114b1d0-ae7f-41b5-b099-044b2cbae49c","Type":"ContainerDied","Data":"e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd"}
Nov 01 20:15:09.045110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.045050    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj" event={"ID":"7114b1d0-ae7f-41b5-b099-044b2cbae49c","Type":"ContainerDied","Data":"e41cb9b029e7456402de422e7de9fa06e5fa632d9a9c6f3fc6c8f9c6d1be1ac3"}
Nov 01 20:15:09.045110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.045070    6479 scope.go:122] "RemoveContainer" containerID="e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd"
Nov 01 20:15:09.045274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.045146    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"
Nov 01 20:15:09.071702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.070616    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" event={"ID":"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82","Type":"ContainerStarted","Data":"75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"}
Nov 01 20:15:09.081257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.081232    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz" event={"ID":"a51d88d1-78c5-48b3-931d-f8d8e3b5b69a","Type":"ContainerDied","Data":"2ef8f0b3ef48ac62cbbfbf119a9fb70d856492fc6acc74d2b83763537c7364ae"}
Nov 01 20:15:09.081519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.081505    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"
Nov 01 20:15:09.084578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.084549    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerStarted","Data":"8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b"}
Nov 01 20:15:09.087412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.087378    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576" event={"ID":"2bf9914b-57ed-4429-b52d-3c2123ad215c","Type":"ContainerDied","Data":"785889d0e577a6e6148b8696a327d49fccb8d56e1f9a51249166b51912b3b3e9"}
Nov 01 20:15:09.087509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.087489    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"
Nov 01 20:15:09.090996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.090932    6479 generic.go:356] "Generic (PLEG): container finished" podID="ee24b5b6-f3d9-4154-8e64-24eaf58f658b" containerID="1fe7a3028b7352beb6e88a07af923bff24217361666606734c5c270e99359244" exitCode=0
Nov 01 20:15:09.090996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.090967    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d" event={"ID":"ee24b5b6-f3d9-4154-8e64-24eaf58f658b","Type":"ContainerDied","Data":"1fe7a3028b7352beb6e88a07af923bff24217361666606734c5c270e99359244"}
Nov 01 20:15:09.091817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.091788    6479 scope.go:122] "RemoveContainer" containerID="e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd"
Nov 01 20:15:09.092195 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:09.092067    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd\": container with ID starting with e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd not found: ID does not exist" containerID="e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd"
Nov 01 20:15:09.092195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.092103    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd"} err="failed to get container status \"e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd\": rpc error: code = NotFound desc = could not find container \"e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd\": container with ID starting with e503db3e8faa8420096f43082678073db96f4885f90471838c79147895d735fd not found: ID does not exist"
Nov 01 20:15:09.092195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.092126    6479 scope.go:122] "RemoveContainer" containerID="c7d806b966f27e7c805ce32c343f08ecd06417eba68824eb644ef66c244bf5f5"
Nov 01 20:15:09.105586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.105568    6479 scope.go:122] "RemoveContainer" containerID="1dd1621442ba3d1e47a14cb01ea1d264d2c9a64eb6e0efbc904e3a8e54ca32f1"
Nov 01 20:15:09.113586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.113531    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podStartSLOduration=3.113516874 podStartE2EDuration="3.113516874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:09.099094622 +0000 UTC m=+954.514727093" watchObservedRunningTime="2026-11-01 20:15:09.113516874 +0000 UTC m=+954.529149347"
Nov 01 20:15:09.115407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.115376    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"]
Nov 01 20:15:09.121852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.121828    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-912c8c21-29892735-x76kj"]
Nov 01 20:15:09.138315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.138288    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv269\" (UniqueName: \"kubernetes.io/projected/7114b1d0-ae7f-41b5-b099-044b2cbae49c-kube-api-access-jv269\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:09.151155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.151130    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"]
Nov 01 20:15:09.156336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.156308    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-912c8c21-78644bcb4f-qp576"]
Nov 01 20:15:09.187569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.187538    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"]
Nov 01 20:15:09.193232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.193201    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-912c8c21-d4b769575-dmsdz"]
Nov 01 20:15:09.547255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.547156    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:09.552743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.552717    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:09.773331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:09.773287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:10.101969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.101935    6479 generic.go:356] "Generic (PLEG): container finished" podID="7f60de4c-c812-4162-8b95-c707b44290a3" containerID="8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b" exitCode=60
Nov 01 20:15:10.102469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.102035    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerDied","Data":"8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b"}
Nov 01 20:15:10.102469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.102099    6479 scope.go:122] "RemoveContainer" containerID="705fb24cff455f6d1b2078864b07dbe63af24176eafe44eaddee742fe3d7b628"
Nov 01 20:15:10.102469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.102344    6479 scope.go:122] "RemoveContainer" containerID="8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b"
Nov 01 20:15:10.102794 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:10.102767    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6_odh-ai-gateway-infra(7f60de4c-c812-4162-8b95-c707b44290a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" podUID="7f60de4c-c812-4162-8b95-c707b44290a3"
Nov 01 20:15:10.104609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.104590    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:10.105752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.105728    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:10.121565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.121546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:10.333285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.333256    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:10.367101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.367015    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w\" (UniqueName: \"kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w\") pod \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") "
Nov 01 20:15:10.367275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.367185    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca\") pod \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") "
Nov 01 20:15:10.381371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.377105    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca\") pod \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\" (UID: \"ee24b5b6-f3d9-4154-8e64-24eaf58f658b\") "
Nov 01 20:15:10.381371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.378949    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca" pod "ee24b5b6-f3d9-4154-8e64-24eaf58f658b" (UID: "ee24b5b6-f3d9-4154-8e64-24eaf58f658b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:10.381371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.379326    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-xks-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:10.381371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.379656    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca" pod "ee24b5b6-f3d9-4154-8e64-24eaf58f658b" (UID: "ee24b5b6-f3d9-4154-8e64-24eaf58f658b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:10.388875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.388845    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w" pod "ee24b5b6-f3d9-4154-8e64-24eaf58f658b" (UID: "ee24b5b6-f3d9-4154-8e64-24eaf58f658b"). InnerVolumeSpecName "kube-api-access-8sr6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:10.481269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.481226    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-maas-api-openshift-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:10.481269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.481265    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sr6w\" (UniqueName: \"kubernetes.io/projected/ee24b5b6-f3d9-4154-8e64-24eaf58f658b-kube-api-access-8sr6w\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:10.528292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.528256    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:10.628779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.628690    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:10.709940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.709906    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf9914b-57ed-4429-b52d-3c2123ad215c" path="/var/lib/kubelet/pods/2bf9914b-57ed-4429-b52d-3c2123ad215c/volumes"
Nov 01 20:15:10.710351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.710338    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7114b1d0-ae7f-41b5-b099-044b2cbae49c" path="/var/lib/kubelet/pods/7114b1d0-ae7f-41b5-b099-044b2cbae49c/volumes"
Nov 01 20:15:10.710743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.710727    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" path="/var/lib/kubelet/pods/a51d88d1-78c5-48b3-931d-f8d8e3b5b69a/volumes"
Nov 01 20:15:10.724457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:10.724423    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:11.118667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.117972    6479 scope.go:122] "RemoveContainer" containerID="8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b"
Nov 01 20:15:11.118667 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:11.118344    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6_odh-ai-gateway-infra(7f60de4c-c812-4162-8b95-c707b44290a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" podUID="7f60de4c-c812-4162-8b95-c707b44290a3"
Nov 01 20:15:11.121895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.121802    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"
Nov 01 20:15:11.121895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.121867    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d" event={"ID":"ee24b5b6-f3d9-4154-8e64-24eaf58f658b","Type":"ContainerDied","Data":"486960130c7f228c96514ac98f31716f9d5c6f789fa231feeb1b7c4815e57f82"}
Nov 01 20:15:11.122105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.121926    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="486960130c7f228c96514ac98f31716f9d5c6f789fa231feeb1b7c4815e57f82"
Nov 01 20:15:11.547869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.547613    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:11.828837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.828566    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:11.912843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.911942    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"]
Nov 01 20:15:11.912843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912794    6479 cpu_manager.go:414] "containerMap: removing container" podUID="2bf9914b-57ed-4429-b52d-3c2123ad215c" containerName="payload-processing"
Nov 01 20:15:11.912843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912814    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf9914b-57ed-4429-b52d-3c2123ad215c" containerName="payload-processing"
Nov 01 20:15:11.912843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912844    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7114b1d0-ae7f-41b5-b099-044b2cbae49c" containerName="cleanup"
Nov 01 20:15:11.912843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912851    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7114b1d0-ae7f-41b5-b099-044b2cbae49c" containerName="cleanup"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912879    6479 cpu_manager.go:414] "containerMap: removing container" podUID="a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" containerName="payload-pre-processing"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912887    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" containerName="payload-pre-processing"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912907    6479 cpu_manager.go:414] "containerMap: removing container" podUID="ee24b5b6-f3d9-4154-8e64-24eaf58f658b" containerName="revoke-keys"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912915    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee24b5b6-f3d9-4154-8e64-24eaf58f658b" containerName="revoke-keys"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912945    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d2473191-76d3-4ca9-a191-cb2fa7f46be1" containerName="payload-processing"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.912953    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2473191-76d3-4ca9-a191-cb2fa7f46be1" containerName="payload-processing"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.913074    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf9914b-57ed-4429-b52d-3c2123ad215c" containerName="payload-processing"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.913086    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7114b1d0-ae7f-41b5-b099-044b2cbae49c" containerName="cleanup"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.913099    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51d88d1-78c5-48b3-931d-f8d8e3b5b69a" containerName="payload-pre-processing"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.913110    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee24b5b6-f3d9-4154-8e64-24eaf58f658b" containerName="revoke-keys"
Nov 01 20:15:11.913260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.913121    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2473191-76d3-4ca9-a191-cb2fa7f46be1" containerName="payload-processing"
Nov 01 20:15:11.922689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.922068    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"]
Nov 01 20:15:11.922689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.922196    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:11.938266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.938181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:11.942548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.942397    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9b54c8e3-dockercfg-gxwzh\""
Nov 01 20:15:11.942548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.942428    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9b54c8e3\""
Nov 01 20:15:11.967260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.967225    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"]
Nov 01 20:15:11.998476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:11.998445    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.038724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.029843    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm\") pod \"payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.038724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.029894    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") " pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.038724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.030025    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9svz\" (UniqueName: \"kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz\") pod \"payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") " pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.038724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.030157    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.081627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.081184    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"]
Nov 01 20:15:12.155980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.155715    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.159899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.159872    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm\") pod \"payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.160346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.160327    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") " pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.161439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.161417    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9svz\" (UniqueName: \"kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz\") pod \"payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") " pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.161930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.161239    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") " pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.162077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.158860    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.334180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.324391    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9svz\" (UniqueName: \"kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz\") pod \"payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") " pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.343050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.343015    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm\") pod \"payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.412461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.412418    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:12.471883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.470668    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:12.575248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.574935    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:12.997907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:12.993878    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"]
Nov 01 20:15:13.300693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:13.300599    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4" event={"ID":"9790e0ee-5ebb-4ef5-9628-228647f67cc7","Type":"ContainerStarted","Data":"3b628117ede9b3c8aaf28870131abd617fd316177df4541326dbdd94fa3b7d25"}
Nov 01 20:15:13.487719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:13.487664    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"]
Nov 01 20:15:13.670691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:13.668733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:14.334683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.331137    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r" event={"ID":"26b5c057-07ee-46dd-8165-7be8413d2d7b","Type":"ContainerStarted","Data":"b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590"}
Nov 01 20:15:14.334683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.331207    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r" event={"ID":"26b5c057-07ee-46dd-8165-7be8413d2d7b","Type":"ContainerStarted","Data":"96a6174ae3b40e627d88d4f9e9d00dbce5a9b4996ec9e1352bdc1bc648a682ae"}
Nov 01 20:15:14.334683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.332223    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:14.372671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.357016    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r" podStartSLOduration=3.356999048 podStartE2EDuration="3.356999048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:14.351508056 +0000 UTC m=+959.767140534" watchObservedRunningTime="2026-11-01 20:15:14.356999048 +0000 UTC m=+959.772631522"
Nov 01 20:15:14.397670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.381622    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4" event={"ID":"9790e0ee-5ebb-4ef5-9628-228647f67cc7","Type":"ContainerStarted","Data":"54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600"}
Nov 01 20:15:14.438669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.409791    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:14.448684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.445567    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4" podStartSLOduration=3.445547066 podStartE2EDuration="3.445547066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:14.44399872 +0000 UTC m=+959.859631195" watchObservedRunningTime="2026-11-01 20:15:14.445547066 +0000 UTC m=+959.861179540"
Nov 01 20:15:14.991426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:14.991369    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:15.665459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:15.665416    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:15.944710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:15.944678    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:16.044860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.044825    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:16.435070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.422923    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"]
Nov 01 20:15:16.435070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.430320    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.458167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.458132    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-85391858a611\""
Nov 01 20:15:16.465079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.465045    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"]
Nov 01 20:15:16.521597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.521558    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.523533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.521618    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.523533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.521682    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85dnm\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.523533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.521766    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.571167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.571135    6479 patch_prober.go:29] interesting pod/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": read tcp 10.134.0.2:47106->10.134.0.84:15021: read: connection reset by peer" start-of-body=
Nov 01 20:15:16.571438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.571403    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" podUID="20f1a937-de00-4cdf-9f95-88311eb793e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": read tcp 10.134.0.2:47106->10.134.0.84:15021: read: connection reset by peer"
Nov 01 20:15:16.623979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.623947    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.624136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.624001    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.624136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.624026    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85dnm\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.624136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.624049    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.624732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.624703    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.624830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.624814    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.626778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.626742    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.632505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.632479    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85dnm\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.679943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.679909    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"]
Nov 01 20:15:16.687176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.686173    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.689721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.689694    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"agc-cred-473fab576e10\""
Nov 01 20:15:16.691978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.691137    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"]
Nov 01 20:15:16.725716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.725632    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.725946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.725923    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.725946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.725953    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.726112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.725974    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77d8h\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.780403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.780374    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:16.830485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.830453    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.830686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.830527    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.830686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.830559    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.830686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.830581    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77d8h\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.831229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.831197    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.831450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.831432    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.834701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.834638    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.835702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.835681    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:15:16.851042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.848698    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77d8h\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:16.916463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.916425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:16.931287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931252    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931286    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931379    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931404    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931532    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931552    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:16.931642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931596    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931669    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931674    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:16.931814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931708    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.931814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931744    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket\") pod \"20f1a937-de00-4cdf-9f95-88311eb793e9\" (UID: \"20f1a937-de00-4cdf-9f95-88311eb793e9\") "
Nov 01 20:15:16.932008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.931900    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:16.932216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932194    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:16.932355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932330    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:16.932524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932507    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:16.932595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932530    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:16.932595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932544    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:16.932595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932557    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f1a937-de00-4cdf-9f95-88311eb793e9-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:16.932595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.932571    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:16.934243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.934212    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:16.934370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.934277    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:15:16.934370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.934306    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "kube-api-access-2fggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:16.934486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:16.934454    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token" pod "20f1a937-de00-4cdf-9f95-88311eb793e9" (UID: "20f1a937-de00-4cdf-9f95-88311eb793e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:17.009257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.008900    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:17.047309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.035893    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:17.047309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.035927    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fggx\" (UniqueName: \"kubernetes.io/projected/20f1a937-de00-4cdf-9f95-88311eb793e9-kube-api-access-2fggx\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:17.047309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.035943    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:17.047309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.035960    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f1a937-de00-4cdf-9f95-88311eb793e9-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:17.076682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.076628    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"]
Nov 01 20:15:17.082041 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:17.082015    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2bb168_0fb3_4e3f_9fad_694b2ad81228.slice/crio-603d1b971578bb5df40d6f30be26944d221418e7cb9b054acf2077f941e0c2ad WatchSource:0}: Error finding container 603d1b971578bb5df40d6f30be26944d221418e7cb9b054acf2077f941e0c2ad: Status 404 returned error can't find the container with id 603d1b971578bb5df40d6f30be26944d221418e7cb9b054acf2077f941e0c2ad
Nov 01 20:15:17.326858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.326832    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"]
Nov 01 20:15:17.328704 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:17.328671    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b56a4f4_c30c_4957_9c90_041964cfd673.slice/crio-50c090c8deef9aa7f52e2b44a187c542ae7a26b57d7a1adf9bd995d0e496ab17 WatchSource:0}: Error finding container 50c090c8deef9aa7f52e2b44a187c542ae7a26b57d7a1adf9bd995d0e496ab17: Status 404 returned error can't find the container with id 50c090c8deef9aa7f52e2b44a187c542ae7a26b57d7a1adf9bd995d0e496ab17
Nov 01 20:15:17.437867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.437833    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97" event={"ID":"5b56a4f4-c30c-4957-9c90-041964cfd673","Type":"ContainerStarted","Data":"eb0c127927dfbbe8411be8b9ca7ea5054443ec7d9ad1d996c0b64e1cbbccdcab"}
Nov 01 20:15:17.438058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.437891    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97" event={"ID":"5b56a4f4-c30c-4957-9c90-041964cfd673","Type":"ContainerStarted","Data":"50c090c8deef9aa7f52e2b44a187c542ae7a26b57d7a1adf9bd995d0e496ab17"}
Nov 01 20:15:17.438058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.438027    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:17.439442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.439414    6479 generic.go:356] "Generic (PLEG): container finished" podID="20f1a937-de00-4cdf-9f95-88311eb793e9" containerID="721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10" exitCode=0
Nov 01 20:15:17.439543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.439461    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"
Nov 01 20:15:17.439543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.439495    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" event={"ID":"20f1a937-de00-4cdf-9f95-88311eb793e9","Type":"ContainerDied","Data":"721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10"}
Nov 01 20:15:17.439622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.439549    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc" event={"ID":"20f1a937-de00-4cdf-9f95-88311eb793e9","Type":"ContainerDied","Data":"e6a6ea1b0186da3bc91229f1dba390705482631a39b133cf91bee6cc9d7a09d1"}
Nov 01 20:15:17.439622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.439580    6479 scope.go:122] "RemoveContainer" containerID="721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10"
Nov 01 20:15:17.441338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.441312    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf" event={"ID":"6b2bb168-0fb3-4e3f-9fad-694b2ad81228","Type":"ContainerStarted","Data":"097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1"}
Nov 01 20:15:17.441460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.441361    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf" event={"ID":"6b2bb168-0fb3-4e3f-9fad-694b2ad81228","Type":"ContainerStarted","Data":"603d1b971578bb5df40d6f30be26944d221418e7cb9b054acf2077f941e0c2ad"}
Nov 01 20:15:17.441460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.441432    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:17.449977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.449962    6479 scope.go:122] "RemoveContainer" containerID="721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10"
Nov 01 20:15:17.450236 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:17.450218    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10\": container with ID starting with 721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10 not found: ID does not exist" containerID="721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10"
Nov 01 20:15:17.450297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.450247    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10"} err="failed to get container status \"721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10\": rpc error: code = NotFound desc = could not find container \"721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10\": container with ID starting with 721de22dee3365d33224c0ef569724d369f76b27cff7165122e5098166c6bb10 not found: ID does not exist"
Nov 01 20:15:17.456148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.456109    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97" podStartSLOduration=1.456097709 podStartE2EDuration="1.456097709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:17.454453239 +0000 UTC m=+962.870085713" watchObservedRunningTime="2026-11-01 20:15:17.456097709 +0000 UTC m=+962.871730179"
Nov 01 20:15:17.475299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.475249    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf" podStartSLOduration=1.475233907 podStartE2EDuration="1.475233907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:17.474820769 +0000 UTC m=+962.890453241" watchObservedRunningTime="2026-11-01 20:15:17.475233907 +0000 UTC m=+962.890866381"
Nov 01 20:15:17.493676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.493630    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"]
Nov 01 20:15:17.496509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.496485    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-912c8c21-openshift-default-78674f977f-ndvnc"]
Nov 01 20:15:17.580056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.579982    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"]
Nov 01 20:15:17.580572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.580558    6479 cpu_manager.go:414] "containerMap: removing container" podUID="20f1a937-de00-4cdf-9f95-88311eb793e9" containerName="istio-proxy"
Nov 01 20:15:17.580630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.580574    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f1a937-de00-4cdf-9f95-88311eb793e9" containerName="istio-proxy"
Nov 01 20:15:17.580706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.580696    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f1a937-de00-4cdf-9f95-88311eb793e9" containerName="istio-proxy"
Nov 01 20:15:17.585526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.585503    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.592888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.592858    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:15:17.593949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.593925    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"]
Nov 01 20:15:17.640096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.640054    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"]
Nov 01 20:15:17.643622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.643575    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58gtp\" (UniqueName: \"kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.643811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.643747    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.644011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.643984    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.657383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.657357    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"]
Nov 01 20:15:17.657558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.657533    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.691315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.691285    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9b54c8e3-serving-cert\""
Nov 01 20:15:17.745332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.745299    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.745527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.745350    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58gtp\" (UniqueName: \"kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.745527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.745381    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.745773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.745746    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnwf\" (UniqueName: \"kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.745901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.745870    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.746003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.745986    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.746050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.746015    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.746096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.746052    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.748170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.748147    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.766446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.766416    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58gtp\" (UniqueName: \"kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp\") pod \"maas-api-revoke-sub-f347a907-ckw8z\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.838629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.838522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:17.848736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.848306    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.849566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.849536    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnwf\" (UniqueName: \"kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.849902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.849877    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.850020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.850005    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.866683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.850937    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.867584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.867554    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.871042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.871006    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnwf\" (UniqueName: \"kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.871814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.871772    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls\") pod \"maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:17.922140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.922099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:17.928245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.927329    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:17.973187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:17.973151    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:18.221372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:18.221345    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"]
Nov 01 20:15:18.223249 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:18.223223    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b467fc_cf1a_4b26_8b22_2ce5041a2ff9.slice/crio-721e22f026cd65b35c07717803fa17bc6bb513594745c7eaa0f3101766515fdc WatchSource:0}: Error finding container 721e22f026cd65b35c07717803fa17bc6bb513594745c7eaa0f3101766515fdc: Status 404 returned error can't find the container with id 721e22f026cd65b35c07717803fa17bc6bb513594745c7eaa0f3101766515fdc
Nov 01 20:15:18.276356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:18.276328    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"]
Nov 01 20:15:18.276701 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:18.276675    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010a0274_2471_49da_943a_f62ce75981fc.slice/crio-c07f3388800fc14070da26c867357f96308f417d2898c7000150d04b4ef5d6ca WatchSource:0}: Error finding container c07f3388800fc14070da26c867357f96308f417d2898c7000150d04b4ef5d6ca: Status 404 returned error can't find the container with id c07f3388800fc14070da26c867357f96308f417d2898c7000150d04b4ef5d6ca
Nov 01 20:15:18.463405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:18.463364    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z" event={"ID":"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9","Type":"ContainerStarted","Data":"721e22f026cd65b35c07717803fa17bc6bb513594745c7eaa0f3101766515fdc"}
Nov 01 20:15:18.465559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:18.465520    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf" event={"ID":"010a0274-2471-49da-943a-f62ce75981fc","Type":"ContainerStarted","Data":"c07f3388800fc14070da26c867357f96308f417d2898c7000150d04b4ef5d6ca"}
Nov 01 20:15:18.652951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:18.652916    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:18.820912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:18.820057    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f1a937-de00-4cdf-9f95-88311eb793e9" path="/var/lib/kubelet/pods/20f1a937-de00-4cdf-9f95-88311eb793e9/volumes"
Nov 01 20:15:19.448467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:19.448433    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:19.473743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:19.473712    6479 generic.go:356] "Generic (PLEG): container finished" podID="a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" containerID="72079bcb40c3e65d9dbc0a2fb7c3a8542c559fac940a02fe1a1664755431dcce" exitCode=0
Nov 01 20:15:19.473922 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:19.473787    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z" event={"ID":"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9","Type":"ContainerDied","Data":"72079bcb40c3e65d9dbc0a2fb7c3a8542c559fac940a02fe1a1664755431dcce"}
Nov 01 20:15:19.475385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:19.475362    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf" event={"ID":"010a0274-2471-49da-943a-f62ce75981fc","Type":"ContainerStarted","Data":"93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72"}
Nov 01 20:15:19.475500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:19.475413    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:19.589431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:19.571164    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf" podStartSLOduration=2.024946616 podStartE2EDuration="2.571138046s" totalImagesPullingTime="546.19143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:19.526064471 +0000 UTC m=+964.941696945" watchObservedRunningTime="2026-11-01 20:15:19.571138046 +0000 UTC m=+964.986770520"
Nov 01 20:15:20.066029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.065991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:20.674223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.674194    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:20.706727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.706691    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca\") pod \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") "
Nov 01 20:15:20.706895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.706756    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp\" (UniqueName: \"kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp\") pod \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") "
Nov 01 20:15:20.706895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.706801    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca\") pod \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\" (UID: \"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9\") "
Nov 01 20:15:20.707249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.707206    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca" pod "a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" (UID: "a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:20.707603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.707578    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-openshift-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:20.714930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.714886    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca" pod "a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" (UID: "a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:20.714930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.714910    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp" pod "a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" (UID: "a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9"). InnerVolumeSpecName "kube-api-access-58gtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:20.763345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.763272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:20.809231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.809197    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-maas-api-xks-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:20.809231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:20.809229    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58gtp\" (UniqueName: \"kubernetes.io/projected/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9-kube-api-access-58gtp\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:21.438344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:21.438309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:21.495358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:21.495322    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z" event={"ID":"a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9","Type":"ContainerDied","Data":"721e22f026cd65b35c07717803fa17bc6bb513594745c7eaa0f3101766515fdc"}
Nov 01 20:15:21.495571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:21.495386    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721e22f026cd65b35c07717803fa17bc6bb513594745c7eaa0f3101766515fdc"
Nov 01 20:15:21.495571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:21.495329    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"
Nov 01 20:15:21.700788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:21.700757    6479 scope.go:122] "RemoveContainer" containerID="8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b"
Nov 01 20:15:21.907302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:21.907268    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:22.049304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:22.049220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:22.504379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:22.504337    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerStarted","Data":"e467c17c7204a63e4f14e292d0aa8580fb9c11b28d471881000f85301621be15"}
Nov 01 20:15:22.713104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:22.713053    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:23.357925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.357882    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:23.518898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.518861    6479 generic.go:356] "Generic (PLEG): container finished" podID="7f60de4c-c812-4162-8b95-c707b44290a3" containerID="e467c17c7204a63e4f14e292d0aa8580fb9c11b28d471881000f85301621be15" exitCode=60
Nov 01 20:15:23.519407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.518939    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerDied","Data":"e467c17c7204a63e4f14e292d0aa8580fb9c11b28d471881000f85301621be15"}
Nov 01 20:15:23.519407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.519020    6479 scope.go:122] "RemoveContainer" containerID="8cb3eb457ff67f5c7b91f44c513d670b252fe6c74ffe873ead2677f30d702e8b"
Nov 01 20:15:23.519524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.519490    6479 scope.go:122] "RemoveContainer" containerID="e467c17c7204a63e4f14e292d0aa8580fb9c11b28d471881000f85301621be15"
Nov 01 20:15:23.519874 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:23.519852    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6_odh-ai-gateway-infra(7f60de4c-c812-4162-8b95-c707b44290a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" podUID="7f60de4c-c812-4162-8b95-c707b44290a3"
Nov 01 20:15:23.529786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.529762    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"]
Nov 01 20:15:23.982428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:23.982382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:24.546076 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.546041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:24.777380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.777315    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:24.793444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.793086    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca\") pod \"7f60de4c-c812-4162-8b95-c707b44290a3\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") "
Nov 01 20:15:24.793444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.793203    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k\" (UniqueName: \"kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k\") pod \"7f60de4c-c812-4162-8b95-c707b44290a3\" (UID: \"7f60de4c-c812-4162-8b95-c707b44290a3\") "
Nov 01 20:15:24.794795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.793885    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca" pod "7f60de4c-c812-4162-8b95-c707b44290a3" (UID: "7f60de4c-c812-4162-8b95-c707b44290a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:24.794795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.794778    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f60de4c-c812-4162-8b95-c707b44290a3-maas-api-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:24.811618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.811583    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k" pod "7f60de4c-c812-4162-8b95-c707b44290a3" (UID: "7f60de4c-c812-4162-8b95-c707b44290a3"). InnerVolumeSpecName "kube-api-access-qhv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:24.896625 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:24.896570    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhv5k\" (UniqueName: \"kubernetes.io/projected/7f60de4c-c812-4162-8b95-c707b44290a3-kube-api-access-qhv5k\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:25.145609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.145513    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:25.279411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.279364    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:15:25.504269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.504238    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:25.546509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.546475    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6" event={"ID":"7f60de4c-c812-4162-8b95-c707b44290a3","Type":"ContainerDied","Data":"b9a57cae6538179a710aadf9dc5a487e0c3fd136ffd30254a80ec3bae8996584"}
Nov 01 20:15:25.551554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.547055    6479 scope.go:122] "RemoveContainer" containerID="e467c17c7204a63e4f14e292d0aa8580fb9c11b28d471881000f85301621be15"
Nov 01 20:15:25.551554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.547197    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"
Nov 01 20:15:25.599418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.599384    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"]
Nov 01 20:15:25.604319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.604046    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892735-f2kj6"]
Nov 01 20:15:25.953420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:25.953356    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:26.465669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:26.459955    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:26.465669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:26.460791    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:26.839150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:26.839075    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" path="/var/lib/kubelet/pods/7f60de4c-c812-4162-8b95-c707b44290a3/volumes"
Nov 01 20:15:26.970731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:26.970676    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:27.984436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:27.984392    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:28.500842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:28.500780    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:15:28.504423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:28.503812    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:15:28.603749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:28.603707    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"]
Nov 01 20:15:28.669693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:28.661730    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"]
Nov 01 20:15:29.385670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:29.353554    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:29.414668 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:29.411106    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:15:30.094669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.093736    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:30.181680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.176498    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:30.226705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.225545    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:30.275463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.249954    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:30.275463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.271947    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:30.565671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.562405    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:30.638769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.638608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:30.814681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:30.814063    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:31.584258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:31.584219    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:32.274670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:32.274251    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:33.076387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:33.076256    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:33.662293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:33.662242    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl" podUID="c1aac181-5c09-4832-b088-0132afbecf31" containerName="payload-processing" containerID="cri-o://ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0" gracePeriod=25
Nov 01 20:15:33.662507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:33.662409    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk" podUID="d8803588-0122-4d3b-9053-f0626f3a14cd" containerName="payload-processing" containerID="cri-o://f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af" gracePeriod=25
Nov 01 20:15:33.850447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:33.848501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:34.056150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.056116    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:15:34.120513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.120416    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:15:34.142145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142109    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials\") pod \"c1aac181-5c09-4832-b088-0132afbecf31\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") "
Nov 01 20:15:34.142145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142156    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume\") pod \"d8803588-0122-4d3b-9053-f0626f3a14cd\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") "
Nov 01 20:15:34.142415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142204    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials\") pod \"d8803588-0122-4d3b-9053-f0626f3a14cd\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") "
Nov 01 20:15:34.142415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142264    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume\") pod \"d8803588-0122-4d3b-9053-f0626f3a14cd\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") "
Nov 01 20:15:34.142415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142338    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc\") pod \"c1aac181-5c09-4832-b088-0132afbecf31\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") "
Nov 01 20:15:34.142415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142386    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt\") pod \"d8803588-0122-4d3b-9053-f0626f3a14cd\" (UID: \"d8803588-0122-4d3b-9053-f0626f3a14cd\") "
Nov 01 20:15:34.142551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142454    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume\") pod \"c1aac181-5c09-4832-b088-0132afbecf31\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") "
Nov 01 20:15:34.142551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.142479    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume\") pod \"c1aac181-5c09-4832-b088-0132afbecf31\" (UID: \"c1aac181-5c09-4832-b088-0132afbecf31\") "
Nov 01 20:15:34.143110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.143076    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume" pod "c1aac181-5c09-4832-b088-0132afbecf31" (UID: "c1aac181-5c09-4832-b088-0132afbecf31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:34.143298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.143271    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume" pod "c1aac181-5c09-4832-b088-0132afbecf31" (UID: "c1aac181-5c09-4832-b088-0132afbecf31"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:34.143596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.143571    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume" pod "d8803588-0122-4d3b-9053-f0626f3a14cd" (UID: "d8803588-0122-4d3b-9053-f0626f3a14cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:34.143976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.143946    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume" pod "d8803588-0122-4d3b-9053-f0626f3a14cd" (UID: "d8803588-0122-4d3b-9053-f0626f3a14cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:34.145721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.145694    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc" pod "c1aac181-5c09-4832-b088-0132afbecf31" (UID: "c1aac181-5c09-4832-b088-0132afbecf31"). InnerVolumeSpecName "kube-api-access-j82nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:34.145912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.145881    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials" pod "c1aac181-5c09-4832-b088-0132afbecf31" (UID: "c1aac181-5c09-4832-b088-0132afbecf31"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:34.146016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.145995    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials" pod "d8803588-0122-4d3b-9053-f0626f3a14cd" (UID: "d8803588-0122-4d3b-9053-f0626f3a14cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:34.146818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.146798    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt" pod "d8803588-0122-4d3b-9053-f0626f3a14cd" (UID: "d8803588-0122-4d3b-9053-f0626f3a14cd"). InnerVolumeSpecName "kube-api-access-xs7gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244143    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j82nc\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-kube-api-access-j82nc\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244189    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7gt\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-kube-api-access-xs7gt\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244204    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244218    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1aac181-5c09-4832-b088-0132afbecf31-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244231    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1aac181-5c09-4832-b088-0132afbecf31-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244246    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244262    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8803588-0122-4d3b-9053-f0626f3a14cd-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.244723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.244275    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8803588-0122-4d3b-9053-f0626f3a14cd-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.345242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.345207    6479 generic.go:356] "Generic (PLEG): container finished" podID="c1aac181-5c09-4832-b088-0132afbecf31" containerID="ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0" exitCode=0
Nov 01 20:15:34.345437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.345371    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"
Nov 01 20:15:34.345805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.345779    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl" event={"ID":"c1aac181-5c09-4832-b088-0132afbecf31","Type":"ContainerDied","Data":"ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0"}
Nov 01 20:15:34.345876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.345835    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl" event={"ID":"c1aac181-5c09-4832-b088-0132afbecf31","Type":"ContainerDied","Data":"031eaa0ef1c79711f3d3196a71751e9e955cecc9b3f5dbd32ca7ee385d59b01a"}
Nov 01 20:15:34.345876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.345866    6479 scope.go:122] "RemoveContainer" containerID="ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0"
Nov 01 20:15:34.348145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.348116    6479 generic.go:356] "Generic (PLEG): container finished" podID="d8803588-0122-4d3b-9053-f0626f3a14cd" containerID="f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af" exitCode=0
Nov 01 20:15:34.348265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.348223    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk" event={"ID":"d8803588-0122-4d3b-9053-f0626f3a14cd","Type":"ContainerDied","Data":"f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af"}
Nov 01 20:15:34.348330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.348263    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk" event={"ID":"d8803588-0122-4d3b-9053-f0626f3a14cd","Type":"ContainerDied","Data":"621c8618d4309f0da4d4df6f305ae4329ffa59a12caec967f09f5ed7786d7209"}
Nov 01 20:15:34.348370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.348332    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"
Nov 01 20:15:34.361145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.361078    6479 scope.go:122] "RemoveContainer" containerID="ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0"
Nov 01 20:15:34.361510 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:34.361484    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0\": container with ID starting with ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0 not found: ID does not exist" containerID="ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0"
Nov 01 20:15:34.361620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.361519    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0"} err="failed to get container status \"ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0\": rpc error: code = NotFound desc = could not find container \"ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0\": container with ID starting with ef88f3fafe304b719c945c4e6dd057bf215f68adabcdb324ee910a09362ff9e0 not found: ID does not exist"
Nov 01 20:15:34.361620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.361547    6479 scope.go:122] "RemoveContainer" containerID="f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af"
Nov 01 20:15:34.383682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.379162    6479 scope.go:122] "RemoveContainer" containerID="f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af"
Nov 01 20:15:34.383682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.379900    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"]
Nov 01 20:15:34.383682 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:34.379961    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af\": container with ID starting with f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af not found: ID does not exist" containerID="f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af"
Nov 01 20:15:34.383682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.379992    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af"} err="failed to get container status \"f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af\": rpc error: code = NotFound desc = could not find container \"f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af\": container with ID starting with f8bb71f20ade7506262827fb62b6b27ae8d31ee1b4832137bf2f084e5a0100af not found: ID does not exist"
Nov 01 20:15:34.383682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.383115    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fflrsl"]
Nov 01 20:15:34.395411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.395088    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"]
Nov 01 20:15:34.398489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.398456    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65khldk"]
Nov 01 20:15:34.478862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.478828    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:34.709510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.709466    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1aac181-5c09-4832-b088-0132afbecf31" path="/var/lib/kubelet/pods/c1aac181-5c09-4832-b088-0132afbecf31/volumes"
Nov 01 20:15:34.710031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:34.710009    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8803588-0122-4d3b-9053-f0626f3a14cd" path="/var/lib/kubelet/pods/d8803588-0122-4d3b-9053-f0626f3a14cd/volumes"
Nov 01 20:15:35.070919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:35.070885    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:35.691466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:35.691393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:36.268481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:36.268448    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:37.052200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.052164    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:37.391423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.391305    6479 generic.go:356] "Generic (PLEG): container finished" podID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerID="75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0" exitCode=0
Nov 01 20:15:37.391423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.391383    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" event={"ID":"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82","Type":"ContainerDied","Data":"75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"}
Nov 01 20:15:37.391852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.391836    6479 scope.go:122] "RemoveContainer" containerID="75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"
Nov 01 20:15:37.393446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.393415    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:37.393534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.393498    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:37.393534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.393531    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:37.768032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:37.767992    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:38.400296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:38.400259    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" event={"ID":"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82","Type":"ContainerStarted","Data":"00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233"}
Nov 01 20:15:38.402157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:38.402135    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:38.558671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:38.554259    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:38.558671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:38.554461    6479 patch_prober.go:29] interesting pod/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 20:15:38.558671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:38.554504    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 20:15:39.043627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:39.043596    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:39.547038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:39.547004    6479 patch_prober.go:29] interesting pod/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 20:15:39.547530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:39.547066    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 20:15:39.633049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:39.633015    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:40.105424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:40.105390    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:40.237740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:40.237706    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:40.547278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:40.547241    6479 patch_prober.go:29] interesting pod/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 20:15:40.547737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:40.547292    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 20:15:40.842779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:40.842690    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:41.543072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:41.543034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:41.546548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:41.546518    6479 patch_prober.go:29] interesting pod/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 20:15:41.546725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:41.546575    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 20:15:42.264756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.260588    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:42.546750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.546640    6479 patch_prober.go:29] interesting pod/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 20:15:42.546750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.546724    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 20:15:42.735371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.735340    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:42.850823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.850732    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:42.899591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.899555    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"]
Nov 01 20:15:42.918815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.918776    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:42.919054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.919032    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:42.922723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.922693    6479 cpu_manager.go:414] "containerMap: removing container" podUID="c1aac181-5c09-4832-b088-0132afbecf31" containerName="payload-processing"
Nov 01 20:15:42.929784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.922883    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1aac181-5c09-4832-b088-0132afbecf31" containerName="payload-processing"
Nov 01 20:15:42.930003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.929984    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d8803588-0122-4d3b-9053-f0626f3a14cd" containerName="payload-processing"
Nov 01 20:15:42.930126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.930112    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8803588-0122-4d3b-9053-f0626f3a14cd" containerName="payload-processing"
Nov 01 20:15:42.930247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.930235    6479 cpu_manager.go:414] "containerMap: removing container" podUID="a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" containerName="revoke-keys"
Nov 01 20:15:42.930345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.930331    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" containerName="revoke-keys"
Nov 01 20:15:42.930523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.930510    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:42.930620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.930610    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:42.930938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.930913    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:42.931067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.931056    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1aac181-5c09-4832-b088-0132afbecf31" containerName="payload-processing"
Nov 01 20:15:42.931179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.931168    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8803588-0122-4d3b-9053-f0626f3a14cd" containerName="payload-processing"
Nov 01 20:15:42.931280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.931270    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" containerName="revoke-keys"
Nov 01 20:15:42.938430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.938391    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:42.989251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.989212    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"]
Nov 01 20:15:42.995770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:42.992006    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.092269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.085544    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjz7\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.092269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.085724    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.092269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.085853    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.092269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.085917    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.180728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.176560    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:43.189132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.189096    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.189455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.189433    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.189616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.189597    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjz7\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.189958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.189929    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.191411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.191376    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.193582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.193552    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.193582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.193560    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.199567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.199545    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjz7\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.435253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.435161    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:43.546996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.546964    6479 patch_prober.go:29] interesting pod/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 20:15:43.547152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.547012    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 20:15:43.832361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.832329    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"]
Nov 01 20:15:43.834579 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:43.834548    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61be557_0a3f_4221_a9be_a9f468b7dd16.slice/crio-d5b1aee6084851c8ae4709326a2b047bf3c6a5a3cfb2e0a6f8d228ef3b7668b3 WatchSource:0}: Error finding container d5b1aee6084851c8ae4709326a2b047bf3c6a5a3cfb2e0a6f8d228ef3b7668b3: Status 404 returned error can't find the container with id d5b1aee6084851c8ae4709326a2b047bf3c6a5a3cfb2e0a6f8d228ef3b7668b3
Nov 01 20:15:43.975495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:43.975456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:44.493493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.493453    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58" event={"ID":"d61be557-0a3f-4221-a9be-a9f468b7dd16","Type":"ContainerStarted","Data":"de83d73cdf77511b32fadd8a05debe742a9c58afc9e42a9949f8c5095047ef16"}
Nov 01 20:15:44.493933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.493521    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58" event={"ID":"d61be557-0a3f-4221-a9be-a9f468b7dd16","Type":"ContainerStarted","Data":"d5b1aee6084851c8ae4709326a2b047bf3c6a5a3cfb2e0a6f8d228ef3b7668b3"}
Nov 01 20:15:44.493933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.493583    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:44.509081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.509029    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58" podStartSLOduration=2.50901113 podStartE2EDuration="2.50901113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:44.50844081 +0000 UTC m=+989.924073283" watchObservedRunningTime="2026-11-01 20:15:44.50901113 +0000 UTC m=+989.924643598"
Nov 01 20:15:44.551201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.551176    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:44.552200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.552166    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:44.630078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:44.630047    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:45.306785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:45.306752    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:45.865791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:45.865751    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:46.122864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:46.122784    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:15:46.455506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:46.455473    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:46.754211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:46.737278    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"]
Nov 01 20:15:46.755606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:46.755567    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"]
Nov 01 20:15:46.793217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:46.793180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:46.893320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:46.893286    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:47.271034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:47.270994    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:48.019480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:48.019345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:48.089293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:48.089260    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"]
Nov 01 20:15:48.115922 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:48.115891    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:48.188014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:48.187984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:48.854969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:48.854934    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:49.736138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:49.736099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:50.016913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.016829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:50.116718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.116673    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:50.492225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.492181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:50.542934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.542896    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"]
Nov 01 20:15:50.543531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.543515    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:50.543531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.543530    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:50.543710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.543629    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f60de4c-c812-4162-8b95-c707b44290a3" containerName="cleanup"
Nov 01 20:15:50.548304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.548286    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.555591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.554143    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a0b8796673b9\""
Nov 01 20:15:50.555591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.554244    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zpkcp\""
Nov 01 20:15:50.560747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.560725    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"]
Nov 01 20:15:50.581620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.581584    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772dl\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.581796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.581668    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.581796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.581736    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.581939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.581800    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.659144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.659105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:50.682868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.682837    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772dl\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.683003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.682888    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.683003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.682962    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.683139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.683039    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.683829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.683798    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.683935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.683866    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.685528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.685500    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.690439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.690417    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-772dl\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl\") pod \"payload-processing-external-model-7c8545db6f-mmvnk\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:50.867159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:50.867072    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:51.043389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.043290    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:51.098565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.098532    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"]
Nov 01 20:15:51.130196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.130062    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:51.376104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.376070    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"]
Nov 01 20:15:51.476838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.442747    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"]
Nov 01 20:15:51.500676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.477278    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf" podUID="010a0274-2471-49da-943a-f62ce75981fc" containerName="maas-api" containerID="cri-o://93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72" gracePeriod=30
Nov 01 20:15:51.502786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.501377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:51.565571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.565537    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk" event={"ID":"acbeaef3-f2eb-497f-b256-6e04896898e4","Type":"ContainerStarted","Data":"b08d766f096757a765154b0de6f3489657ea1edf85396a4c028b0b8c2f47aae0"}
Nov 01 20:15:51.768967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.749220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:51.771091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.770918    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4" podUID="9790e0ee-5ebb-4ef5-9628-228647f67cc7" containerName="payload-processing" containerID="cri-o://54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600" gracePeriod=25
Nov 01 20:15:51.771387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.771279    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r" podUID="26b5c057-07ee-46dd-8165-7be8413d2d7b" containerName="payload-pre-processing" containerID="cri-o://b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590" gracePeriod=25
Nov 01 20:15:51.859942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.859907    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:51.949883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:51.948440    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:52.048695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.048185    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:52.052745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.052723    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:52.097380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.096865    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls\") pod \"010a0274-2471-49da-943a-f62ce75981fc\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") "
Nov 01 20:15:52.097380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.096918    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca\") pod \"010a0274-2471-49da-943a-f62ce75981fc\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") "
Nov 01 20:15:52.097380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.097021    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs\") pod \"010a0274-2471-49da-943a-f62ce75981fc\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") "
Nov 01 20:15:52.097380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.097076    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf\" (UniqueName: \"kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf\") pod \"010a0274-2471-49da-943a-f62ce75981fc\" (UID: \"010a0274-2471-49da-943a-f62ce75981fc\") "
Nov 01 20:15:52.100133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.099269    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca" pod "010a0274-2471-49da-943a-f62ce75981fc" (UID: "010a0274-2471-49da-943a-f62ce75981fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:52.102088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.101856    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf" pod "010a0274-2471-49da-943a-f62ce75981fc" (UID: "010a0274-2471-49da-943a-f62ce75981fc"). InnerVolumeSpecName "kube-api-access-6gnwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:52.102088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.101960    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls" pod "010a0274-2471-49da-943a-f62ce75981fc" (UID: "010a0274-2471-49da-943a-f62ce75981fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:52.102263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.102132    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs" pod "010a0274-2471-49da-943a-f62ce75981fc" (UID: "010a0274-2471-49da-943a-f62ce75981fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:52.177514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.177477    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:52.198610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.198580    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.198798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.198612    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gnwf\" (UniqueName: \"kubernetes.io/projected/010a0274-2471-49da-943a-f62ce75981fc-kube-api-access-6gnwf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.198798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.198631    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010a0274-2471-49da-943a-f62ce75981fc-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.198798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.198657    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010a0274-2471-49da-943a-f62ce75981fc-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.199019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.198831    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:52.204871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.204836    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:52.243688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.243636    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"]
Nov 01 20:15:52.244680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.244527    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9790e0ee-5ebb-4ef5-9628-228647f67cc7" containerName="payload-processing"
Nov 01 20:15:52.244680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.244550    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9790e0ee-5ebb-4ef5-9628-228647f67cc7" containerName="payload-processing"
Nov 01 20:15:52.244680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.244568    6479 cpu_manager.go:414] "containerMap: removing container" podUID="26b5c057-07ee-46dd-8165-7be8413d2d7b" containerName="payload-pre-processing"
Nov 01 20:15:52.244680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.244577    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b5c057-07ee-46dd-8165-7be8413d2d7b" containerName="payload-pre-processing"
Nov 01 20:15:52.245160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.244765    6479 cpu_manager.go:414] "containerMap: removing container" podUID="010a0274-2471-49da-943a-f62ce75981fc" containerName="maas-api"
Nov 01 20:15:52.245160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.244781    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010a0274-2471-49da-943a-f62ce75981fc" containerName="maas-api"
Nov 01 20:15:52.245160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.245040    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9790e0ee-5ebb-4ef5-9628-228647f67cc7" containerName="payload-processing"
Nov 01 20:15:52.245160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.245058    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b5c057-07ee-46dd-8165-7be8413d2d7b" containerName="payload-pre-processing"
Nov 01 20:15:52.245160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.245082    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="010a0274-2471-49da-943a-f62ce75981fc" containerName="maas-api"
Nov 01 20:15:52.248956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.248936    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.269549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.269478    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hvvr7\""
Nov 01 20:15:52.274577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.272827    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"]
Nov 01 20:15:52.299485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.299439    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume\") pod \"26b5c057-07ee-46dd-8165-7be8413d2d7b\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") "
Nov 01 20:15:52.299661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.299624    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume\") pod \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") "
Nov 01 20:15:52.299749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.299704    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm\") pod \"26b5c057-07ee-46dd-8165-7be8413d2d7b\" (UID: \"26b5c057-07ee-46dd-8165-7be8413d2d7b\") "
Nov 01 20:15:52.299812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.299773    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz\" (UniqueName: \"kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz\") pod \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\" (UID: \"9790e0ee-5ebb-4ef5-9628-228647f67cc7\") "
Nov 01 20:15:52.300211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.300185    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume" pod "9790e0ee-5ebb-4ef5-9628-228647f67cc7" (UID: "9790e0ee-5ebb-4ef5-9628-228647f67cc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:52.300398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.300371    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.300398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.300388    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume" pod "26b5c057-07ee-46dd-8165-7be8413d2d7b" (UID: "26b5c057-07ee-46dd-8165-7be8413d2d7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:52.300527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.300483    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncjv\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.300682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.300659    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.300959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.300912    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.301126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.301101    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b5c057-07ee-46dd-8165-7be8413d2d7b-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.301181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.301144    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9790e0ee-5ebb-4ef5-9628-228647f67cc7-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.302331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.302292    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm" pod "26b5c057-07ee-46dd-8165-7be8413d2d7b" (UID: "26b5c057-07ee-46dd-8165-7be8413d2d7b"). InnerVolumeSpecName "kube-api-access-g8bwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:52.303437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.303417    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz" pod "9790e0ee-5ebb-4ef5-9628-228647f67cc7" (UID: "9790e0ee-5ebb-4ef5-9628-228647f67cc7"). InnerVolumeSpecName "kube-api-access-j9svz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:52.404672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.404538    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.404882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.404713    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.404882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.404771    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncjv\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.404882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.404866    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.405088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.404942    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bwm\" (UniqueName: \"kubernetes.io/projected/26b5c057-07ee-46dd-8165-7be8413d2d7b-kube-api-access-g8bwm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.405088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.404958    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9svz\" (UniqueName: \"kubernetes.io/projected/9790e0ee-5ebb-4ef5-9628-228647f67cc7-kube-api-access-j9svz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:52.415668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.405748    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.477724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.476752    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.481668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.478149    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.481668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.478560    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:52.516210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.514616    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"]
Nov 01 20:15:52.516210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.514889    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy" containerID="cri-o://00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233" gracePeriod=30
Nov 01 20:15:52.620665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.536675    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncjv\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv\") pod \"payload-processing-external-model-7c8545db6f-llqbv\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") " pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.620665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.615134    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:52.676099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.669635    6479 generic.go:356] "Generic (PLEG): container finished" podID="26b5c057-07ee-46dd-8165-7be8413d2d7b" containerID="b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590" exitCode=0
Nov 01 20:15:52.676099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.669774    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r" event={"ID":"26b5c057-07ee-46dd-8165-7be8413d2d7b","Type":"ContainerDied","Data":"b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590"}
Nov 01 20:15:52.676099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.669828    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r" event={"ID":"26b5c057-07ee-46dd-8165-7be8413d2d7b","Type":"ContainerDied","Data":"96a6174ae3b40e627d88d4f9e9d00dbce5a9b4996ec9e1352bdc1bc648a682ae"}
Nov 01 20:15:52.676099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.669856    6479 scope.go:122] "RemoveContainer" containerID="b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590"
Nov 01 20:15:52.676099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.670029    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"
Nov 01 20:15:52.699044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.681032    6479 generic.go:356] "Generic (PLEG): container finished" podID="9790e0ee-5ebb-4ef5-9628-228647f67cc7" containerID="54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600" exitCode=0
Nov 01 20:15:52.699044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.681129    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4" event={"ID":"9790e0ee-5ebb-4ef5-9628-228647f67cc7","Type":"ContainerDied","Data":"54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600"}
Nov 01 20:15:52.699044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.681180    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4" event={"ID":"9790e0ee-5ebb-4ef5-9628-228647f67cc7","Type":"ContainerDied","Data":"3b628117ede9b3c8aaf28870131abd617fd316177df4541326dbdd94fa3b7d25"}
Nov 01 20:15:52.699044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.681279    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"
Nov 01 20:15:52.699044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.697971    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk" event={"ID":"acbeaef3-f2eb-497f-b256-6e04896898e4","Type":"ContainerStarted","Data":"311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8"}
Nov 01 20:15:52.699044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.698215    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:52.733690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.714580    6479 generic.go:356] "Generic (PLEG): container finished" podID="010a0274-2471-49da-943a-f62ce75981fc" containerID="93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72" exitCode=0
Nov 01 20:15:52.733690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.714788    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"
Nov 01 20:15:52.768829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.766383    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk" podStartSLOduration=2.766363776 podStartE2EDuration="2.766363776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:52.764590104 +0000 UTC m=+998.180222579" watchObservedRunningTime="2026-11-01 20:15:52.766363776 +0000 UTC m=+998.181996251"
Nov 01 20:15:52.768829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.767464    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf" event={"ID":"010a0274-2471-49da-943a-f62ce75981fc","Type":"ContainerDied","Data":"93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72"}
Nov 01 20:15:52.768829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.767506    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf" event={"ID":"010a0274-2471-49da-943a-f62ce75981fc","Type":"ContainerDied","Data":"c07f3388800fc14070da26c867357f96308f417d2898c7000150d04b4ef5d6ca"}
Nov 01 20:15:52.851762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.851713    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"]
Nov 01 20:15:52.879695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.858708    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b54c8e3-79f574d7f9-4gm8r"]
Nov 01 20:15:52.879695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.872832    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"]
Nov 01 20:15:52.893810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.893729    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b54c8e3-57b46bcf65-sjjjf"]
Nov 01 20:15:52.898014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.897985    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"]
Nov 01 20:15:52.900774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.900750    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b54c8e3-5c67cb5cc-nxxc4"]
Nov 01 20:15:52.900953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.900924    6479 scope.go:122] "RemoveContainer" containerID="b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590"
Nov 01 20:15:52.901352 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:52.901331    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590\": container with ID starting with b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590 not found: ID does not exist" containerID="b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590"
Nov 01 20:15:52.901447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.901364    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590"} err="failed to get container status \"b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590\": rpc error: code = NotFound desc = could not find container \"b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590\": container with ID starting with b6cbb4fcf9cb2284486e5ea39e687b1056f033b302c3e21e3bc3003ac69d8590 not found: ID does not exist"
Nov 01 20:15:52.901447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.901391    6479 scope.go:122] "RemoveContainer" containerID="54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600"
Nov 01 20:15:52.934857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.934824    6479 scope.go:122] "RemoveContainer" containerID="54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600"
Nov 01 20:15:52.935537 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:52.935374    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600\": container with ID starting with 54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600 not found: ID does not exist" containerID="54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600"
Nov 01 20:15:52.935537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.935425    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600"} err="failed to get container status \"54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600\": rpc error: code = NotFound desc = could not find container \"54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600\": container with ID starting with 54916ea1f39ffa56fa0bd0f55d8f1d1881ead79ee0a43f9460607303686f8600 not found: ID does not exist"
Nov 01 20:15:52.935537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:52.935450    6479 scope.go:122] "RemoveContainer" containerID="93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72"
Nov 01 20:15:53.065025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.064957    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:53.066436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.066316    6479 scope.go:122] "RemoveContainer" containerID="93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72"
Nov 01 20:15:53.066730 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:53.066706    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72\": container with ID starting with 93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72 not found: ID does not exist" containerID="93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72"
Nov 01 20:15:53.066795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.066741    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72"} err="failed to get container status \"93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72\": rpc error: code = NotFound desc = could not find container \"93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72\": container with ID starting with 93700a63330591a53e6292a6e193f4a482f950189101adf44462ed42ee9ffa72 not found: ID does not exist"
Nov 01 20:15:53.090588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.090549    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf" podUID="de615f26-cdfa-48b5-acb4-5d6765ca3e6a" containerName="payload-processing" containerID="cri-o://faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce" gracePeriod=25
Nov 01 20:15:53.160146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.160116    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:53.187795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.187767    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"]
Nov 01 20:15:53.190680 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:53.190620    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb471174b_2796_4f9c_9a5f_c659f5484216.slice/crio-16ac5ac07ed3eae021ddf31356b84a84b7123271c6735c687e0a8ad280295609 WatchSource:0}: Error finding container 16ac5ac07ed3eae021ddf31356b84a84b7123271c6735c687e0a8ad280295609: Status 404 returned error can't find the container with id 16ac5ac07ed3eae021ddf31356b84a84b7123271c6735c687e0a8ad280295609
Nov 01 20:15:53.254306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.254274    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:53.274925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.274890    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"]
Nov 01 20:15:53.359242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.359094    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:53.419110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.419079    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:15:53.452534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.452504    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"]
Nov 01 20:15:53.453198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.453180    6479 cpu_manager.go:414] "containerMap: removing container" podUID="de615f26-cdfa-48b5-acb4-5d6765ca3e6a" containerName="payload-processing"
Nov 01 20:15:53.453198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.453199    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de615f26-cdfa-48b5-acb4-5d6765ca3e6a" containerName="payload-processing"
Nov 01 20:15:53.453356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.453311    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="de615f26-cdfa-48b5-acb4-5d6765ca3e6a" containerName="payload-processing"
Nov 01 20:15:53.456726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.456700    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.471244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.468259    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ea421713-openshift-default-dockercfg-rj6pf\""
Nov 01 20:15:53.471244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.470917    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"]
Nov 01 20:15:53.497831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.497798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:53.530027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.529998    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume\") pod \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") "
Nov 01 20:15:53.530217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530117    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5\") pod \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") "
Nov 01 20:15:53.530280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530237    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume\") pod \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") "
Nov 01 20:15:53.530337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530304    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials\") pod \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\" (UID: \"de615f26-cdfa-48b5-acb4-5d6765ca3e6a\") "
Nov 01 20:15:53.530389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530336    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume" pod "de615f26-cdfa-48b5-acb4-5d6765ca3e6a" (UID: "de615f26-cdfa-48b5-acb4-5d6765ca3e6a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:53.530634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530608    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume" pod "de615f26-cdfa-48b5-acb4-5d6765ca3e6a" (UID: "de615f26-cdfa-48b5-acb4-5d6765ca3e6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:53.530817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530802    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:53.530881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.530821    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:53.532594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.532566    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5" pod "de615f26-cdfa-48b5-acb4-5d6765ca3e6a" (UID: "de615f26-cdfa-48b5-acb4-5d6765ca3e6a"). InnerVolumeSpecName "kube-api-access-knft5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:53.532750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.532623    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials" pod "de615f26-cdfa-48b5-acb4-5d6765ca3e6a" (UID: "de615f26-cdfa-48b5-acb4-5d6765ca3e6a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632667    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632720    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632787    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632814    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632843    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632867    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632932    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.632984    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.633048    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrz4x\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.633366    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:53.636752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.633386    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knft5\" (UniqueName: \"kubernetes.io/projected/de615f26-cdfa-48b5-acb4-5d6765ca3e6a-kube-api-access-knft5\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:53.672670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.665853    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737722    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737769    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737805    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737829    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737858    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737879    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737934    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.737979    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.738045    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrz4x\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.738935    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.740407    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.746528    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.750106    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.760576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.750345    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.773987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.773896    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.796617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.796585    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.800665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.797082    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrz4x\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.814561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.814243    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo\") pod \"e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") " pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:53.828457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.828328    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv" event={"ID":"b471174b-2796-4f9c-9a5f-c659f5484216","Type":"ContainerStarted","Data":"0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2"}
Nov 01 20:15:53.828457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.828394    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv" event={"ID":"b471174b-2796-4f9c-9a5f-c659f5484216","Type":"ContainerStarted","Data":"16ac5ac07ed3eae021ddf31356b84a84b7123271c6735c687e0a8ad280295609"}
Nov 01 20:15:53.828457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.828423    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:53.857819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.856934    6479 generic.go:356] "Generic (PLEG): container finished" podID="de615f26-cdfa-48b5-acb4-5d6765ca3e6a" containerID="faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce" exitCode=0
Nov 01 20:15:53.857819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.857090    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf" event={"ID":"de615f26-cdfa-48b5-acb4-5d6765ca3e6a","Type":"ContainerDied","Data":"faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce"}
Nov 01 20:15:53.857819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.857180    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf" event={"ID":"de615f26-cdfa-48b5-acb4-5d6765ca3e6a","Type":"ContainerDied","Data":"404a09c9b6200d3e5040bfb363982413b1fdbca9d9cd72aeade4591517dadb15"}
Nov 01 20:15:53.857819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.857213    6479 scope.go:122] "RemoveContainer" containerID="faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce"
Nov 01 20:15:53.857819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.857321    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"
Nov 01 20:15:53.876799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.876739    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv" podStartSLOduration=1.876720405 podStartE2EDuration="1.876720405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:53.869153818 +0000 UTC m=+999.284786292" watchObservedRunningTime="2026-11-01 20:15:53.876720405 +0000 UTC m=+999.292352879"
Nov 01 20:15:53.927993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.927969    6479 scope.go:122] "RemoveContainer" containerID="faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce"
Nov 01 20:15:53.928474 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:53.928446    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce\": container with ID starting with faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce not found: ID does not exist" containerID="faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce"
Nov 01 20:15:53.928563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.928484    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce"} err="failed to get container status \"faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce\": rpc error: code = NotFound desc = could not find container \"faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce\": container with ID starting with faf5fa55f7a4678d1d02171bfe7fc6d92b5506089e22331ebf007da772cc15ce not found: ID does not exist"
Nov 01 20:15:53.941433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.941405    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"]
Nov 01 20:15:53.946550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:53.946520    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-566dcd79c7-n9nvf"]
Nov 01 20:15:54.072468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.072432    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:54.080095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.080066    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:54.376267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.376237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:54.496724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.494239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:54.498670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.498032    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"]
Nov 01 20:15:54.530672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.525831    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"]
Nov 01 20:15:54.605966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.578600    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:54.605966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.579102    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:54.605966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.579507    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:15:54.682248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.680725    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"]
Nov 01 20:15:54.729459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.729329    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:15:54.729459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.729397    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.729716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.729402    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"]
Nov 01 20:15:54.740134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.740106    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010a0274-2471-49da-943a-f62ce75981fc" path="/var/lib/kubelet/pods/010a0274-2471-49da-943a-f62ce75981fc/volumes"
Nov 01 20:15:54.740556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.740541    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b5c057-07ee-46dd-8165-7be8413d2d7b" path="/var/lib/kubelet/pods/26b5c057-07ee-46dd-8165-7be8413d2d7b/volumes"
Nov 01 20:15:54.740953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.740938    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9790e0ee-5ebb-4ef5-9628-228647f67cc7" path="/var/lib/kubelet/pods/9790e0ee-5ebb-4ef5-9628-228647f67cc7/volumes"
Nov 01 20:15:54.741576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.741561    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de615f26-cdfa-48b5-acb4-5d6765ca3e6a" path="/var/lib/kubelet/pods/de615f26-cdfa-48b5-acb4-5d6765ca3e6a/volumes"
Nov 01 20:15:54.790760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.790680    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.790901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.790783    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6k5\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.790901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.790818    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.791109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.791083    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.917697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.914512    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.917697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.914671    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.917697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.914767    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6k5\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.917697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.914808    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.917697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.916097    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.917697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.916717    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:54.986381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.969206    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6k5\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:55.001219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.995124    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" event={"ID":"09fdd51b-5f92-4df4-8cc6-40eef12c0378","Type":"ContainerStarted","Data":"90c8dcd6709c9495591b2c8e025725f68e686888d616fb8a9fc22a77870f48cc"}
Nov 01 20:15:55.001219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:54.995790    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:55.023733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.022591    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:55.153633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.153548    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:55.190378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.190314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:55.311776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.311533    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:55.519698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.519622    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"]
Nov 01 20:15:55.570509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.565832    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:55.574877 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:55.574842    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5871aa_a1e0_4ad9_8c3e_08dd4edc946e.slice/crio-4855d95a1e9d056b53a7ed51598f09c75e3f47745fdf07d68a6a5df282fcb5aa WatchSource:0}: Error finding container 4855d95a1e9d056b53a7ed51598f09c75e3f47745fdf07d68a6a5df282fcb5aa: Status 404 returned error can't find the container with id 4855d95a1e9d056b53a7ed51598f09c75e3f47745fdf07d68a6a5df282fcb5aa
Nov 01 20:15:55.603756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.603702    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"]
Nov 01 20:15:55.612466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.612438    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"]
Nov 01 20:15:55.612593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.612554    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.618938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.616587    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa9f1c40297b\""
Nov 01 20:15:55.653444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.653416    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.653715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.653690    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.653823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.653742    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.653823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.653772    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnlc\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.762206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.755150    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.762206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.755382    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.762206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.755431    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.762206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.755460    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnlc\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.762206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.759689    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.762206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.760330    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.769159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.768547    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.841013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.840936    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnlc\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc\") pod \"payload-processing-external-model-598fcbc94f-mgqwx\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") " pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:55.933939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:55.933899    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:15:56.082190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.082148    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p" event={"ID":"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e","Type":"ContainerStarted","Data":"a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5"}
Nov 01 20:15:56.082380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.082230    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p" event={"ID":"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e","Type":"ContainerStarted","Data":"4855d95a1e9d056b53a7ed51598f09c75e3f47745fdf07d68a6a5df282fcb5aa"}
Nov 01 20:15:56.082771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.082751    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:15:56.085044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.085019    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" event={"ID":"09fdd51b-5f92-4df4-8cc6-40eef12c0378","Type":"ContainerStarted","Data":"191402052ef6d892fea7db67f6074ea7ad5a5866758ae2b393b151bf9722519a"}
Nov 01 20:15:56.108971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.107474    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p" podStartSLOduration=2.107454958 podStartE2EDuration="2.107454958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:56.104824413 +0000 UTC m=+1001.520456900" watchObservedRunningTime="2026-11-01 20:15:56.107454958 +0000 UTC m=+1001.523087432"
Nov 01 20:15:56.145402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.145371    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:56.185047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.184982    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" podStartSLOduration=3.184962644 podStartE2EDuration="3.184962644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:56.131868917 +0000 UTC m=+1001.547501393" watchObservedRunningTime="2026-11-01 20:15:56.184962644 +0000 UTC m=+1001.600595119"
Nov 01 20:15:56.187231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.187201    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"]
Nov 01 20:15:56.247360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.247314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:56.269050 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:15:56.269015    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a9f996_a755_47df_b386_4c29ec8fe3da.slice/crio-cd1918589373ab6f8f6c599dc8612c4fce92c81dc25302780755184743c13bf5 WatchSource:0}: Error finding container cd1918589373ab6f8f6c599dc8612c4fce92c81dc25302780755184743c13bf5: Status 404 returned error can't find the container with id cd1918589373ab6f8f6c599dc8612c4fce92c81dc25302780755184743c13bf5
Nov 01 20:15:56.270133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:56.270106    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"]
Nov 01 20:15:57.080383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.080347    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:57.085961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.085930    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:57.095623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.095589    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx" event={"ID":"64a9f996-a755-47df-b386-4c29ec8fe3da","Type":"ContainerStarted","Data":"dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f"}
Nov 01 20:15:57.095795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.095670    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx" event={"ID":"64a9f996-a755-47df-b386-4c29ec8fe3da","Type":"ContainerStarted","Data":"cd1918589373ab6f8f6c599dc8612c4fce92c81dc25302780755184743c13bf5"}
Nov 01 20:15:57.095923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.095901    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:57.096878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.096856    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:15:57.140699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.140622    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx" podStartSLOduration=2.140604483 podStartE2EDuration="2.140604483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:57.138986242 +0000 UTC m=+1002.554618716" watchObservedRunningTime="2026-11-01 20:15:57.140604483 +0000 UTC m=+1002.556236956"
Nov 01 20:15:57.248085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.248055    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:57.650381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.650351    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:57.698777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.698730    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk" podUID="acbeaef3-f2eb-497f-b256-6e04896898e4" containerName="payload-processing" containerID="cri-o://311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8" gracePeriod=25
Nov 01 20:15:57.757169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.757086    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:57.832952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.832915    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:57.927994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.927953    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.928349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.928285    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.928445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.928398    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.928445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.928395    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:57.928958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.928937    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.929072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929024    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.929139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929082    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.929196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929145    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.929252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929227    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.929314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929258    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data\") pod \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\" (UID: \"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82\") "
Nov 01 20:15:57.929942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929818    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:57.929942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929759    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:57.929942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.929892    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:57.930194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.930094    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:57.930774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.930747    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:57.930877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.930805    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:57.930877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.930822    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:57.930877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.930836    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:57.930877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.930868    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:57.933154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.933129    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "kube-api-access-4mpq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:57.934036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.934010    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:57.934358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.934328    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:15:57.935698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.935640    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo" pod "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" (UID: "7cc54af2-3c7a-45f9-99a1-90ed1c03bd82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:15:57.978016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:57.977993    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:58.032068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032040    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume\") pod \"acbeaef3-f2eb-497f-b256-6e04896898e4\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") "
Nov 01 20:15:58.032252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032146    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl\") pod \"acbeaef3-f2eb-497f-b256-6e04896898e4\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") "
Nov 01 20:15:58.032252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032167    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials\") pod \"acbeaef3-f2eb-497f-b256-6e04896898e4\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") "
Nov 01 20:15:58.032252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032195    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume\") pod \"acbeaef3-f2eb-497f-b256-6e04896898e4\" (UID: \"acbeaef3-f2eb-497f-b256-6e04896898e4\") "
Nov 01 20:15:58.032478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032441    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume" pod "acbeaef3-f2eb-497f-b256-6e04896898e4" (UID: "acbeaef3-f2eb-497f-b256-6e04896898e4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:58.032692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032674    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.032769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032696    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpq4\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-kube-api-access-4mpq4\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.032769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032711    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.032769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032715    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume" pod "acbeaef3-f2eb-497f-b256-6e04896898e4" (UID: "acbeaef3-f2eb-497f-b256-6e04896898e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:58.032769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032724    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.032769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.032765    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.034329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.034308    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl" pod "acbeaef3-f2eb-497f-b256-6e04896898e4" (UID: "acbeaef3-f2eb-497f-b256-6e04896898e4"). InnerVolumeSpecName "kube-api-access-772dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:58.034425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.034353    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials" pod "acbeaef3-f2eb-497f-b256-6e04896898e4" (UID: "acbeaef3-f2eb-497f-b256-6e04896898e4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:58.043169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.043147    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:58.146940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.146841    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772dl\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-kube-api-access-772dl\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.146940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.146877    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/acbeaef3-f2eb-497f-b256-6e04896898e4-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.146940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.146892    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbeaef3-f2eb-497f-b256-6e04896898e4-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.163569    6479 generic.go:356] "Generic (PLEG): container finished" podID="acbeaef3-f2eb-497f-b256-6e04896898e4" containerID="311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8" exitCode=0
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.163748    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk" event={"ID":"acbeaef3-f2eb-497f-b256-6e04896898e4","Type":"ContainerDied","Data":"311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8"}
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.163798    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk" event={"ID":"acbeaef3-f2eb-497f-b256-6e04896898e4","Type":"ContainerDied","Data":"b08d766f096757a765154b0de6f3489657ea1edf85396a4c028b0b8c2f47aae0"}
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.163828    6479 scope.go:122] "RemoveContainer" containerID="311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8"
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.164006    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.170513    6479 generic.go:356] "Generic (PLEG): container finished" podID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerID="00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233" exitCode=0
Nov 01 20:15:58.171671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.171447    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"
Nov 01 20:15:58.175201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.173882    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" event={"ID":"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82","Type":"ContainerDied","Data":"00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233"}
Nov 01 20:15:58.175201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.173956    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd" event={"ID":"7cc54af2-3c7a-45f9-99a1-90ed1c03bd82","Type":"ContainerDied","Data":"e171f466792bcfc0ec1ac25937f632bb33f521a1a21ffc976990b17df48e290c"}
Nov 01 20:15:58.208102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.208080    6479 scope.go:122] "RemoveContainer" containerID="311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8"
Nov 01 20:15:58.208479 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:58.208456    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8\": container with ID starting with 311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8 not found: ID does not exist" containerID="311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8"
Nov 01 20:15:58.208552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.208486    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8"} err="failed to get container status \"311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8\": rpc error: code = NotFound desc = could not find container \"311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8\": container with ID starting with 311de5148e3e568e68d43e108d6188feafb5e7c0c38df341e061fa79e5fa84c8 not found: ID does not exist"
Nov 01 20:15:58.208552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.208504    6479 scope.go:122] "RemoveContainer" containerID="00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233"
Nov 01 20:15:58.216640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.216617    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"]
Nov 01 20:15:58.219098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.219079    6479 scope.go:122] "RemoveContainer" containerID="75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"
Nov 01 20:15:58.221021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.221001    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9b54c8e3-openshift-default-5bc4dbb5f5-l9xdd"]
Nov 01 20:15:58.228372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.228352    6479 scope.go:122] "RemoveContainer" containerID="00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233"
Nov 01 20:15:58.228657 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:58.228627    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233\": container with ID starting with 00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233 not found: ID does not exist" containerID="00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233"
Nov 01 20:15:58.228716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.228665    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233"} err="failed to get container status \"00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233\": rpc error: code = NotFound desc = could not find container \"00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233\": container with ID starting with 00838d90312b4452e6c4c10773989800affb5c690d9c823fec136446b6678233 not found: ID does not exist"
Nov 01 20:15:58.228716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.228685    6479 scope.go:122] "RemoveContainer" containerID="75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"
Nov 01 20:15:58.228921 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:58.228906    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0\": container with ID starting with 75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0 not found: ID does not exist" containerID="75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"
Nov 01 20:15:58.228962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.228926    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0"} err="failed to get container status \"75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0\": rpc error: code = NotFound desc = could not find container \"75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0\": container with ID starting with 75d1c053648ac4010d9f56ab59611c2e45efc1e3e39d22eae13100cc164693f0 not found: ID does not exist"
Nov 01 20:15:58.235369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.235350    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"]
Nov 01 20:15:58.238999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.238976    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-mmvnk"]
Nov 01 20:15:58.737635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.737600    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" path="/var/lib/kubelet/pods/7cc54af2-3c7a-45f9-99a1-90ed1c03bd82/volumes"
Nov 01 20:15:58.738802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.738777    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbeaef3-f2eb-497f-b256-6e04896898e4" path="/var/lib/kubelet/pods/acbeaef3-f2eb-497f-b256-6e04896898e4/volumes"
Nov 01 20:15:58.832826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.832767    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv" podUID="b471174b-2796-4f9c-9a5f-c659f5484216" containerName="payload-processing" containerID="cri-o://0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2" gracePeriod=25
Nov 01 20:15:58.979254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:58.979222    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:59.088896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.088864    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"]
Nov 01 20:15:59.089448 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089435    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy"
Nov 01 20:15:59.089493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089449    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy"
Nov 01 20:15:59.089493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089467    6479 cpu_manager.go:414] "containerMap: removing container" podUID="acbeaef3-f2eb-497f-b256-6e04896898e4" containerName="payload-processing"
Nov 01 20:15:59.089493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089473    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbeaef3-f2eb-497f-b256-6e04896898e4" containerName="payload-processing"
Nov 01 20:15:59.089597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089576    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy"
Nov 01 20:15:59.089597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089589    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy"
Nov 01 20:15:59.089668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.089597    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbeaef3-f2eb-497f-b256-6e04896898e4" containerName="payload-processing"
Nov 01 20:15:59.093061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.093040    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.109006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.108985    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ea421713\""
Nov 01 20:15:59.115134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.115101    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ea421713-dockercfg-t8nlr\""
Nov 01 20:15:59.119574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.119537    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"]
Nov 01 20:15:59.120283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.120263    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy"
Nov 01 20:15:59.120402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.120385    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc54af2-3c7a-45f9-99a1-90ed1c03bd82" containerName="istio-proxy"
Nov 01 20:15:59.124740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.124722    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"]
Nov 01 20:15:59.124854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.124816    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.143332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.143308    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:59.146462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.146430    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"]
Nov 01 20:15:59.180388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.179673    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d257l\" (UniqueName: \"kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l\") pod \"payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.180388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.179735    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume\") pod \"payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") " pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.180388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.179770    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.180388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.179820    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmk7\" (UniqueName: \"kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7\") pod \"payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") " pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.183790 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:59.183756    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:15:59.184642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.184620    6479 generic.go:356] "Generic (PLEG): container finished" podID="b471174b-2796-4f9c-9a5f-c659f5484216" containerID="0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2" exitCode=0
Nov 01 20:15:59.184772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.184754    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"
Nov 01 20:15:59.184874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.184852    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv" event={"ID":"b471174b-2796-4f9c-9a5f-c659f5484216","Type":"ContainerDied","Data":"0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2"}
Nov 01 20:15:59.185005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.184989    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv" event={"ID":"b471174b-2796-4f9c-9a5f-c659f5484216","Type":"ContainerDied","Data":"16ac5ac07ed3eae021ddf31356b84a84b7123271c6735c687e0a8ad280295609"}
Nov 01 20:15:59.185139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.185123    6479 scope.go:122] "RemoveContainer" containerID="0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2"
Nov 01 20:15:59.197133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.197107    6479 scope.go:122] "RemoveContainer" containerID="0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2"
Nov 01 20:15:59.197499 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:15:59.197472    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2\": container with ID starting with 0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2 not found: ID does not exist" containerID="0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2"
Nov 01 20:15:59.197616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.197511    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2"} err="failed to get container status \"0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2\": rpc error: code = NotFound desc = could not find container \"0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2\": container with ID starting with 0c88e71be0caed3c6cf5ac66583fc180308066ef1319f5cec905290ea9944ce2 not found: ID does not exist"
Nov 01 20:15:59.226222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.226187    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.287181    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv\") pod \"b471174b-2796-4f9c-9a5f-c659f5484216\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") "
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.287247    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume\") pod \"b471174b-2796-4f9c-9a5f-c659f5484216\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") "
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.287384    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials\") pod \"b471174b-2796-4f9c-9a5f-c659f5484216\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") "
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.287446    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume\") pod \"b471174b-2796-4f9c-9a5f-c659f5484216\" (UID: \"b471174b-2796-4f9c-9a5f-c659f5484216\") "
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.288046    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d257l\" (UniqueName: \"kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l\") pod \"payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.288114    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume\") pod \"payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") " pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.288169    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.294721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.288255    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmk7\" (UniqueName: \"kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7\") pod \"payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") " pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.316029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.300693    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume" pod "b471174b-2796-4f9c-9a5f-c659f5484216" (UID: "b471174b-2796-4f9c-9a5f-c659f5484216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:59.345946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.328062    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume" pod "b471174b-2796-4f9c-9a5f-c659f5484216" (UID: "b471174b-2796-4f9c-9a5f-c659f5484216"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:59.345946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.332374    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:15:59.345946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.333330    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.345946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.344335    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume\") pod \"payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") " pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.365479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.357799    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials" pod "b471174b-2796-4f9c-9a5f-c659f5484216" (UID: "b471174b-2796-4f9c-9a5f-c659f5484216"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:59.390680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.388877    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmk7\" (UniqueName: \"kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7\") pod \"payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") " pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.390880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.390684    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:59.390880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.390715    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:59.390880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.390731    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b471174b-2796-4f9c-9a5f-c659f5484216-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:59.415898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.415357    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d257l\" (UniqueName: \"kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l\") pod \"payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.415898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.415845    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv" pod "b471174b-2796-4f9c-9a5f-c659f5484216" (UID: "b471174b-2796-4f9c-9a5f-c659f5484216"). InnerVolumeSpecName "kube-api-access-vncjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:59.436700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.434508    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:15:59.470673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.464011    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:15:59.504986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.503599    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vncjv\" (UniqueName: \"kubernetes.io/projected/b471174b-2796-4f9c-9a5f-c659f5484216-kube-api-access-vncjv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:59.548288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.548165    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58" podUID="d61be557-0a3f-4221-a9be-a9f468b7dd16" containerName="payload-processing" containerID="cri-o://de83d73cdf77511b32fadd8a05debe742a9c58afc9e42a9949f8c5095047ef16" gracePeriod=25
Nov 01 20:15:59.748806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.701504    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"]
Nov 01 20:15:59.756148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.751976    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c8545db6f-llqbv"]
Nov 01 20:15:59.777828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:15:59.777775    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:00.307743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:00.307370    6479 generic.go:356] "Generic (PLEG): container finished" podID="d61be557-0a3f-4221-a9be-a9f468b7dd16" containerID="de83d73cdf77511b32fadd8a05debe742a9c58afc9e42a9949f8c5095047ef16" exitCode=0
Nov 01 20:16:00.307743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:00.307493    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58" event={"ID":"d61be557-0a3f-4221-a9be-a9f468b7dd16","Type":"ContainerDied","Data":"de83d73cdf77511b32fadd8a05debe742a9c58afc9e42a9949f8c5095047ef16"}
Nov 01 20:16:00.470908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:00.442761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:00.749351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:00.749251    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:01.399734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.377508    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b471174b-2796-4f9c-9a5f-c659f5484216" path="/var/lib/kubelet/pods/b471174b-2796-4f9c-9a5f-c659f5484216/volumes"
Nov 01 20:16:01.489685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.458333    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"]
Nov 01 20:16:01.489685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.458408    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"]
Nov 01 20:16:01.587275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.587243    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" event={"ID":"f3cb43f7-16f1-476b-af8d-ba77b9ddf770","Type":"ContainerStarted","Data":"406f8fccd34c3604562e4d133a7503b5fd9f5cf08141065326868c954c0709cc"}
Nov 01 20:16:01.634989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.624259    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr" event={"ID":"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a","Type":"ContainerStarted","Data":"76ae025f7068e3cbd526a0adfd2a0d96d6f95e87ac7f6fc8de65c110741e26d8"}
Nov 01 20:16:01.717681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.693364    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:16:01.782668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.782480    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume\") pod \"d61be557-0a3f-4221-a9be-a9f468b7dd16\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") "
Nov 01 20:16:01.782668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.782542    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials\") pod \"d61be557-0a3f-4221-a9be-a9f468b7dd16\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") "
Nov 01 20:16:01.782668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.782641    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume\") pod \"d61be557-0a3f-4221-a9be-a9f468b7dd16\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") "
Nov 01 20:16:01.783226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.782833    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7\") pod \"d61be557-0a3f-4221-a9be-a9f468b7dd16\" (UID: \"d61be557-0a3f-4221-a9be-a9f468b7dd16\") "
Nov 01 20:16:01.813912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.800607    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume" pod "d61be557-0a3f-4221-a9be-a9f468b7dd16" (UID: "d61be557-0a3f-4221-a9be-a9f468b7dd16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:01.824679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.822102    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume" pod "d61be557-0a3f-4221-a9be-a9f468b7dd16" (UID: "d61be557-0a3f-4221-a9be-a9f468b7dd16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:01.892674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.888962    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7" pod "d61be557-0a3f-4221-a9be-a9f468b7dd16" (UID: "d61be557-0a3f-4221-a9be-a9f468b7dd16"). InnerVolumeSpecName "kube-api-access-jjjz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:01.896183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.896143    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjz7\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-kube-api-access-jjjz7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:01.921285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.919737    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:01.921537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.921516    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61be557-0a3f-4221-a9be-a9f468b7dd16-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:01.982839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:01.982795    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials" pod "d61be557-0a3f-4221-a9be-a9f468b7dd16" (UID: "d61be557-0a3f-4221-a9be-a9f468b7dd16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:02.028363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.028316    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d61be557-0a3f-4221-a9be-a9f468b7dd16-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:02.098179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.096761    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx" podUID="64a9f996-a755-47df-b386-4c29ec8fe3da" containerName="payload-processing" containerID="cri-o://dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f" gracePeriod=25
Nov 01 20:16:02.144534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.144496    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:02.447461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.447428    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:16:02.538263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.538183    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume\") pod \"64a9f996-a755-47df-b386-4c29ec8fe3da\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") "
Nov 01 20:16:02.538263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.538220    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume\") pod \"64a9f996-a755-47df-b386-4c29ec8fe3da\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") "
Nov 01 20:16:02.538497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.538297    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc\") pod \"64a9f996-a755-47df-b386-4c29ec8fe3da\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") "
Nov 01 20:16:02.538497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.538417    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials\") pod \"64a9f996-a755-47df-b386-4c29ec8fe3da\" (UID: \"64a9f996-a755-47df-b386-4c29ec8fe3da\") "
Nov 01 20:16:02.538738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.538711    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume" pod "64a9f996-a755-47df-b386-4c29ec8fe3da" (UID: "64a9f996-a755-47df-b386-4c29ec8fe3da"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:02.538820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.538752    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume" pod "64a9f996-a755-47df-b386-4c29ec8fe3da" (UID: "64a9f996-a755-47df-b386-4c29ec8fe3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:02.539225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.539197    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:02.539321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.539228    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64a9f996-a755-47df-b386-4c29ec8fe3da-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:02.540940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.540912    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc" pod "64a9f996-a755-47df-b386-4c29ec8fe3da" (UID: "64a9f996-a755-47df-b386-4c29ec8fe3da"). InnerVolumeSpecName "kube-api-access-ljnlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:02.541420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.541405    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials" pod "64a9f996-a755-47df-b386-4c29ec8fe3da" (UID: "64a9f996-a755-47df-b386-4c29ec8fe3da"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:02.645919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.640839    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnlc\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-kube-api-access-ljnlc\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:02.645919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.640882    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64a9f996-a755-47df-b386-4c29ec8fe3da-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:02.670677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.665725    6479 generic.go:356] "Generic (PLEG): container finished" podID="64a9f996-a755-47df-b386-4c29ec8fe3da" containerID="dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f" exitCode=0
Nov 01 20:16:02.670677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.665930    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx" event={"ID":"64a9f996-a755-47df-b386-4c29ec8fe3da","Type":"ContainerDied","Data":"dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f"}
Nov 01 20:16:02.670677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.666028    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx" event={"ID":"64a9f996-a755-47df-b386-4c29ec8fe3da","Type":"ContainerDied","Data":"cd1918589373ab6f8f6c599dc8612c4fce92c81dc25302780755184743c13bf5"}
Nov 01 20:16:02.670677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.666059    6479 scope.go:122] "RemoveContainer" containerID="dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f"
Nov 01 20:16:02.670677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.666230    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"
Nov 01 20:16:02.683626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.673112    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" event={"ID":"f3cb43f7-16f1-476b-af8d-ba77b9ddf770","Type":"ContainerStarted","Data":"d255956879cae64dde4df102c2cbd3e3bcc64dce0836eed88ed222d3ccb560ef"}
Nov 01 20:16:02.683626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.673262    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:16:02.683626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.676636    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr" event={"ID":"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a","Type":"ContainerStarted","Data":"67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1"}
Nov 01 20:16:02.683626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.677384    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:16:02.684213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.684099    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58" event={"ID":"d61be557-0a3f-4221-a9be-a9f468b7dd16","Type":"ContainerDied","Data":"d5b1aee6084851c8ae4709326a2b047bf3c6a5a3cfb2e0a6f8d228ef3b7668b3"}
Nov 01 20:16:02.684213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.684200    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"
Nov 01 20:16:02.694180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.694125    6479 scope.go:122] "RemoveContainer" containerID="dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f"
Nov 01 20:16:02.694746 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:02.694714    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f\": container with ID starting with dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f not found: ID does not exist" containerID="dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f"
Nov 01 20:16:02.694746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.694756    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f"} err="failed to get container status \"dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f\": rpc error: code = NotFound desc = could not find container \"dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f\": container with ID starting with dd3cb509c79622631ddb8e8ccc4e5466779b30be6c65b794aa90e87950cf9c1f not found: ID does not exist"
Nov 01 20:16:02.694746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.694782    6479 scope.go:122] "RemoveContainer" containerID="de83d73cdf77511b32fadd8a05debe742a9c58afc9e42a9949f8c5095047ef16"
Nov 01 20:16:02.697217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.697174    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" podStartSLOduration=3.69715899 podStartE2EDuration="3.69715899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:02.69438174 +0000 UTC m=+1008.110014279" watchObservedRunningTime="2026-11-01 20:16:02.69715899 +0000 UTC m=+1008.112791464"
Nov 01 20:16:02.715829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.715787    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr" podStartSLOduration=3.7157772529999997 podStartE2EDuration="3.715777253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:02.714788023 +0000 UTC m=+1008.130420496" watchObservedRunningTime="2026-11-01 20:16:02.715777253 +0000 UTC m=+1008.131409726"
Nov 01 20:16:02.733068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.733041    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"]
Nov 01 20:16:02.736973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.736949    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-785cgzb58"]
Nov 01 20:16:02.749103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.749072    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"]
Nov 01 20:16:02.751997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:02.751977    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-598fcbc94f-mgqwx"]
Nov 01 20:16:03.039852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:03.039819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:03.939992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:03.939953    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:04.554732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:04.554699    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:04.652233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:04.652198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:04.781672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:04.779574    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a9f996-a755-47df-b386-4c29ec8fe3da" path="/var/lib/kubelet/pods/64a9f996-a755-47df-b386-4c29ec8fe3da/volumes"
Nov 01 20:16:04.781672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:04.780297    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61be557-0a3f-4221-a9be-a9f468b7dd16" path="/var/lib/kubelet/pods/d61be557-0a3f-4221-a9be-a9f468b7dd16/volumes"
Nov 01 20:16:05.222109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.222076    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:05.322141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.322107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:05.737967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.737937    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"]
Nov 01 20:16:05.738948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.738928    6479 cpu_manager.go:414] "containerMap: removing container" podUID="64a9f996-a755-47df-b386-4c29ec8fe3da" containerName="payload-processing"
Nov 01 20:16:05.739042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739033    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a9f996-a755-47df-b386-4c29ec8fe3da" containerName="payload-processing"
Nov 01 20:16:05.739116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739108    6479 cpu_manager.go:414] "containerMap: removing container" podUID="b471174b-2796-4f9c-9a5f-c659f5484216" containerName="payload-processing"
Nov 01 20:16:05.739162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739155    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b471174b-2796-4f9c-9a5f-c659f5484216" containerName="payload-processing"
Nov 01 20:16:05.739227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739220    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d61be557-0a3f-4221-a9be-a9f468b7dd16" containerName="payload-processing"
Nov 01 20:16:05.739274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739268    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61be557-0a3f-4221-a9be-a9f468b7dd16" containerName="payload-processing"
Nov 01 20:16:05.739429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739419    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a9f996-a755-47df-b386-4c29ec8fe3da" containerName="payload-processing"
Nov 01 20:16:05.739491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739484    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="b471174b-2796-4f9c-9a5f-c659f5484216" containerName="payload-processing"
Nov 01 20:16:05.739548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.739542    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61be557-0a3f-4221-a9be-a9f468b7dd16" containerName="payload-processing"
Nov 01 20:16:05.744460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.744430    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.745085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.745064    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"]
Nov 01 20:16:05.754048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.754026    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-fec91d8db4b3\""
Nov 01 20:16:05.770038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.770013    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"]
Nov 01 20:16:05.770337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.770321    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"]
Nov 01 20:16:05.770465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.770259    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.805878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.805848    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.806030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.805911    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.806030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.805935    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.806030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.805960    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.806030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.805979    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.806208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.806072    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.806208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.806109    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stx7v\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.806208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.806160    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2nn\" (UniqueName: \"kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.835836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.835803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:05.854009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.853983    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ea421713-serving-cert\""
Nov 01 20:16:05.907014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.906985    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.907152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907039    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stx7v\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.907152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907089    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2nn\" (UniqueName: \"kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.907152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907134    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.907330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907176    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.907330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907200    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.907330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907232    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.907330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907256    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.907844 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:05.907818    6479 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-serving-cert: secret "maas-api-e2e-derive-ea421713-serving-cert" not found
Nov 01 20:16:05.907844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.907833    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.908032 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:05.907898    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls podName:f6004cd0-426b-4be9-bf90-2a1b7f4be179 nodeName:}" failed. No retries permitted until 2026-11-01 20:16:06.407880079 +0000 UTC m=+1011.823512547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls") pod "maas-api-e2e-derive-ea421713-696f87c995-jg92f" (UID: "f6004cd0-426b-4be9-bf90-2a1b7f4be179") : secret "maas-api-e2e-derive-ea421713-serving-cert" not found
Nov 01 20:16:05.908101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.908047    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.908163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.908116    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.909781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.909757    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.909867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.909820    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.914830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.914806    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2nn\" (UniqueName: \"kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:05.914923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.914855    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stx7v\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:05.918075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:05.918045    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:06.040202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.035277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:06.084412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.084382    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:06.265246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.265210    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"]
Nov 01 20:16:06.271059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.271037    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.275623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.275596    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"]
Nov 01 20:16:06.314693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.314580    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.314693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.314623    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwkp\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.314886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.314806    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.314886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.314865    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.341022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.340986    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"agc-cred-a9325faf1444\""
Nov 01 20:16:06.366993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.366963    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"]
Nov 01 20:16:06.367561 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:06.367532    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711ea31a_e508_4ae9_abe7_28a4755776b0.slice/crio-cfefafed2524a9c9e4bd1a5e67dccd4c4d0b3235c2464c13dd396735a202dd92 WatchSource:0}: Error finding container cfefafed2524a9c9e4bd1a5e67dccd4c4d0b3235c2464c13dd396735a202dd92: Status 404 returned error can't find the container with id cfefafed2524a9c9e4bd1a5e67dccd4c4d0b3235c2464c13dd396735a202dd92
Nov 01 20:16:06.416472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.416435    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.416607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.416494    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwkp\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.416607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.416558    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.416607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.416598    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.416799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.416730    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:06.417252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.417224    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.417455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.417423    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.420405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.420351    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.420405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.420395    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\") pod \"maas-api-e2e-derive-ea421713-696f87c995-jg92f\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:06.424632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.424612    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwkp\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.589712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.589625    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:06.686405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.686360    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:06.693665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.693396    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:06.714094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.714061    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"]
Nov 01 20:16:06.726865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.725001    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.726865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.725638    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"]
Nov 01 20:16:06.730623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.730597    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9de2ac-dockercfg-85nj9\""
Nov 01 20:16:06.731018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.731001    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9de2ac\""
Nov 01 20:16:06.741061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.741032    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"]
Nov 01 20:16:06.753679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.753637    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.761511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.761479    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck" event={"ID":"711ea31a-e508-4ae9-abe7-28a4755776b0","Type":"ContainerStarted","Data":"3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d"}
Nov 01 20:16:06.761745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.761723    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck" event={"ID":"711ea31a-e508-4ae9-abe7-28a4755776b0","Type":"ContainerStarted","Data":"cfefafed2524a9c9e4bd1a5e67dccd4c4d0b3235c2464c13dd396735a202dd92"}
Nov 01 20:16:06.762218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.762175    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:06.778688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.773857    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"]
Nov 01 20:16:06.779570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.779536    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:06.829393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.829343    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.829545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.829494    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.829812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.829779    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfd9g\" (UniqueName: \"kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g\") pod \"payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.829918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.829848    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wp2q\" (UniqueName: \"kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q\") pod \"payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.880995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.880967    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:06.882887 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:06.882351    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fff25d_3468_48e2_b697_cc404a2c8c8f.slice/crio-80ae59141f0b8327268fec80159a95c040c169401cdca3cd5c0c6e5e862bfd84 WatchSource:0}: Error finding container 80ae59141f0b8327268fec80159a95c040c169401cdca3cd5c0c6e5e862bfd84: Status 404 returned error can't find the container with id 80ae59141f0b8327268fec80159a95c040c169401cdca3cd5c0c6e5e862bfd84
Nov 01 20:16:06.882887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.882543    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck" podStartSLOduration=1.882524368 podStartE2EDuration="1.882524368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:06.797729743 +0000 UTC m=+1012.213362217" watchObservedRunningTime="2026-11-01 20:16:06.882524368 +0000 UTC m=+1012.298156842"
Nov 01 20:16:06.883390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.883368    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"]
Nov 01 20:16:06.888242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.887859    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"]
Nov 01 20:16:06.890063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.890034    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"]
Nov 01 20:16:06.920509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.920485    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:16:06.931205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.931176    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.931509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.931241    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.931509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.931418    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfd9g\" (UniqueName: \"kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g\") pod \"payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.931509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.931458    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wp2q\" (UniqueName: \"kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q\") pod \"payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.932199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.932175    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.934230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.934208    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.939299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.939280    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wp2q\" (UniqueName: \"kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q\") pod \"payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:06.939604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.939586    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfd9g\" (UniqueName: \"kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g\") pod \"payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:06.976489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:06.976452    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:07.014895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.014855    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"]
Nov 01 20:16:07.015200 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:07.015173    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6004cd0_426b_4be9_bf90_2a1b7f4be179.slice/crio-81a0348ca7b4279ff13a02ce941a920a24320afee693f0bc1330aef0bf77c019 WatchSource:0}: Error finding container 81a0348ca7b4279ff13a02ce941a920a24320afee693f0bc1330aef0bf77c019: Status 404 returned error can't find the container with id 81a0348ca7b4279ff13a02ce941a920a24320afee693f0bc1330aef0bf77c019
Nov 01 20:16:07.053884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.053852    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:07.083562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.083530    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:07.137239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.127452    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"]
Nov 01 20:16:07.137239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.134287    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.139911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.139885    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-8d9fe7\"/\"default-dockercfg-lw2rw\""
Nov 01 20:16:07.140071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.139993    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-8d9fe7\"/\"kube-root-ca.crt\""
Nov 01 20:16:07.140152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.140090    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-8d9fe7\"/\"e2e-usage-sim-71a622-cert\""
Nov 01 20:16:07.140218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.140165    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-8d9fe7\"/\"openshift-service-ca.crt\""
Nov 01 20:16:07.143551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.143517    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"]
Nov 01 20:16:07.166594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.166568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:07.243033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.242528    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7fh\" (UniqueName: \"kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh\") pod \"e2e-usage-sim-71a622-65f44b5db4-9597g\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") " pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.243033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.242736    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls\") pod \"e2e-usage-sim-71a622-65f44b5db4-9597g\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") " pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.324202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.324168    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:07.351771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.351540    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls\") pod \"e2e-usage-sim-71a622-65f44b5db4-9597g\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") " pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.433133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.433091    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7fh\" (UniqueName: \"kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh\") pod \"e2e-usage-sim-71a622-65f44b5db4-9597g\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") " pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.434909 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:07.410072    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bba70c6_6a6b_4cf7_9ac2_bd64f727f5f4.slice/crio-b628028ba897d5ef6e26178d18610d6c8cd664c817da92feb468504b166cc1b2 WatchSource:0}: Error finding container b628028ba897d5ef6e26178d18610d6c8cd664c817da92feb468504b166cc1b2: Status 404 returned error can't find the container with id b628028ba897d5ef6e26178d18610d6c8cd664c817da92feb468504b166cc1b2
Nov 01 20:16:07.470244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.464572    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"]
Nov 01 20:16:07.563591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.563561    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls\") pod \"e2e-usage-sim-71a622-65f44b5db4-9597g\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") " pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.579443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.579330    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7fh\" (UniqueName: \"kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh\") pod \"e2e-usage-sim-71a622-65f44b5db4-9597g\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") " pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.635455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.635309    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"]
Nov 01 20:16:07.757121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.757088    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:07.772537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.772500    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf" event={"ID":"4cae4a8b-1f12-403c-a008-9a271813df87","Type":"ContainerStarted","Data":"eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286"}
Nov 01 20:16:07.772688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.772568    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf" event={"ID":"4cae4a8b-1f12-403c-a008-9a271813df87","Type":"ContainerStarted","Data":"b841102967adcb1c372a2864f59584d416de4ac96aed3615cb479e01075d428f"}
Nov 01 20:16:07.772855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.772835    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:07.775688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.775519    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f" event={"ID":"f6004cd0-426b-4be9-bf90-2a1b7f4be179","Type":"ContainerStarted","Data":"ca500f39b681c054b689475337cdf89af7b725eb97d7f68af9ddc12f0a305a18"}
Nov 01 20:16:07.775688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.775573    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f" event={"ID":"f6004cd0-426b-4be9-bf90-2a1b7f4be179","Type":"ContainerStarted","Data":"81a0348ca7b4279ff13a02ce941a920a24320afee693f0bc1330aef0bf77c019"}
Nov 01 20:16:07.775688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.775603    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:07.777314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.777291    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd" event={"ID":"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4","Type":"ContainerStarted","Data":"e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022"}
Nov 01 20:16:07.777415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.777339    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd" event={"ID":"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4","Type":"ContainerStarted","Data":"b628028ba897d5ef6e26178d18610d6c8cd664c817da92feb468504b166cc1b2"}
Nov 01 20:16:07.777415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.777383    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:07.779314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.779281    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz" event={"ID":"d2fff25d-3468-48e2-b697-cc404a2c8c8f","Type":"ContainerStarted","Data":"4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d"}
Nov 01 20:16:07.780778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.779359    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz" event={"ID":"d2fff25d-3468-48e2-b697-cc404a2c8c8f","Type":"ContainerStarted","Data":"80ae59141f0b8327268fec80159a95c040c169401cdca3cd5c0c6e5e862bfd84"}
Nov 01 20:16:07.780778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.779431    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:07.789187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.789133    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf" podStartSLOduration=1.789114379 podStartE2EDuration="1.789114379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:07.787925837 +0000 UTC m=+1013.203558311" watchObservedRunningTime="2026-11-01 20:16:07.789114379 +0000 UTC m=+1013.204746852"
Nov 01 20:16:07.808538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.808480    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz" podStartSLOduration=1.808459477 podStartE2EDuration="1.808459477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:07.80338672 +0000 UTC m=+1013.219019195" watchObservedRunningTime="2026-11-01 20:16:07.808459477 +0000 UTC m=+1013.224091953"
Nov 01 20:16:07.840584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.840515    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f" podStartSLOduration=2.262576939 podStartE2EDuration="2.840495379s" totalImagesPullingTime="577.91844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:07.822043285 +0000 UTC m=+1013.237675774" watchObservedRunningTime="2026-11-01 20:16:07.840495379 +0000 UTC m=+1013.256127852"
Nov 01 20:16:07.844413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.844356    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd" podStartSLOduration=1.844343556 podStartE2EDuration="1.844343556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:07.839910081 +0000 UTC m=+1013.255542554" watchObservedRunningTime="2026-11-01 20:16:07.844343556 +0000 UTC m=+1013.259976029"
Nov 01 20:16:07.971105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:07.971072    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:08.099060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.097832    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"]
Nov 01 20:16:08.648499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.648407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:08.789202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.789158    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g" event={"ID":"211ddc12-81e0-4492-a7ac-32458b2f6f62","Type":"ContainerStarted","Data":"cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910"}
Nov 01 20:16:08.789729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.789705    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:08.789795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.789749    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g" event={"ID":"211ddc12-81e0-4492-a7ac-32458b2f6f62","Type":"ContainerStarted","Data":"76c1dd3298ae479c25bc3c4867086c144cfb2009cdda73eb40a777f46294cec9"}
Nov 01 20:16:08.794472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.794447    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:16:08.805391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:08.805343    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g" podStartSLOduration=1.805329504 podStartE2EDuration="1.805329504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:08.802936196 +0000 UTC m=+1014.218568670" watchObservedRunningTime="2026-11-01 20:16:08.805329504 +0000 UTC m=+1014.220961976"
Nov 01 20:16:09.355230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:09.355196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:10.008781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:10.008738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:10.742370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:10.742336    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:11.307896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.307865    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"]
Nov 01 20:16:11.312982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.312960    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.322012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.321972    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-9de2ac-serving-cert\""
Nov 01 20:16:11.326673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.326629    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.326799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.326768    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phf9m\" (UniqueName: \"kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.326870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.326840    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.327122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.327093    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.330598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.330572    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"]
Nov 01 20:16:11.361311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.361280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:11.428001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.427964    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phf9m\" (UniqueName: \"kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.428172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.428025    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.428172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.428129    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.428266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.428171    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.428726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.428696    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.430544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.430513    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.430692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.430552    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.442898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.442872    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phf9m\" (UniqueName: \"kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m\") pod \"maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.454664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.454350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:11.558750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.554344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:11.645809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.645775    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:11.889217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.889130    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg" podUID="01c2fad9-a87d-47ca-b6b9-2df730d1d529" containerName="payload-processing" containerID="cri-o://b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102" gracePeriod=25
Nov 01 20:16:11.890624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.890565    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p" podUID="9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" containerName="payload-processing" containerID="cri-o://a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5" gracePeriod=25
Nov 01 20:16:11.932854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:11.932826    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"]
Nov 01 20:16:12.067670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.066583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:12.347506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.347478    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:16:12.402451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.402102    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.476754    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume\") pod \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.476870    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials\") pod \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.476935    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz\") pod \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.476992    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume\") pod \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.477027    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5\") pod \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.477075    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume\") pod \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.477105    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume\") pod \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\" (UID: \"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e\") "
Nov 01 20:16:12.478074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.477313    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials\") pod \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\" (UID: \"01c2fad9-a87d-47ca-b6b9-2df730d1d529\") "
Nov 01 20:16:12.478572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.478230    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume" pod "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" (UID: "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:12.478572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.478380    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume" pod "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" (UID: "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:12.478572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.478516    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume" pod "01c2fad9-a87d-47ca-b6b9-2df730d1d529" (UID: "01c2fad9-a87d-47ca-b6b9-2df730d1d529"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:12.479723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.479696    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5" pod "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" (UID: "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e"). InnerVolumeSpecName "kube-api-access-7r6k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:12.479886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.479847    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume" pod "01c2fad9-a87d-47ca-b6b9-2df730d1d529" (UID: "01c2fad9-a87d-47ca-b6b9-2df730d1d529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:12.480541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.480515    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz" pod "01c2fad9-a87d-47ca-b6b9-2df730d1d529" (UID: "01c2fad9-a87d-47ca-b6b9-2df730d1d529"). InnerVolumeSpecName "kube-api-access-82dkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:12.485338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.485311    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials" pod "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" (UID: "9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:12.486548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.486526    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials" pod "01c2fad9-a87d-47ca-b6b9-2df730d1d529" (UID: "01c2fad9-a87d-47ca-b6b9-2df730d1d529"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:12.579480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579448    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579479    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579489    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579498    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c2fad9-a87d-47ca-b6b9-2df730d1d529-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579509    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579522    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82dkz\" (UniqueName: \"kubernetes.io/projected/01c2fad9-a87d-47ca-b6b9-2df730d1d529-kube-api-access-82dkz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579536    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.579717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.579545    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6k5\" (UniqueName: \"kubernetes.io/projected/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e-kube-api-access-7r6k5\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:12.767433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.767322    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:12.906670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.899014    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m" event={"ID":"66b12b04-38d7-4fbd-8b9c-b3de31c6225c","Type":"ContainerStarted","Data":"54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61"}
Nov 01 20:16:12.906670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.899083    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m" event={"ID":"66b12b04-38d7-4fbd-8b9c-b3de31c6225c","Type":"ContainerStarted","Data":"b3f88e12f139952d90645d71c9eeaaa11d7e709a9a443e5728b7e11a76bf96fa"}
Nov 01 20:16:12.906670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.900162    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:12.930672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.930565    6479 generic.go:356] "Generic (PLEG): container finished" podID="9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" containerID="a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5" exitCode=0
Nov 01 20:16:12.930853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.930734    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p" event={"ID":"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e","Type":"ContainerDied","Data":"a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5"}
Nov 01 20:16:12.930853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.930792    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p" event={"ID":"9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e","Type":"ContainerDied","Data":"4855d95a1e9d056b53a7ed51598f09c75e3f47745fdf07d68a6a5df282fcb5aa"}
Nov 01 20:16:12.930853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.930825    6479 scope.go:122] "RemoveContainer" containerID="a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5"
Nov 01 20:16:12.931019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.930995    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"
Nov 01 20:16:12.944702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.944637    6479 generic.go:356] "Generic (PLEG): container finished" podID="01c2fad9-a87d-47ca-b6b9-2df730d1d529" containerID="b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102" exitCode=0
Nov 01 20:16:12.947897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.944850    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg" event={"ID":"01c2fad9-a87d-47ca-b6b9-2df730d1d529","Type":"ContainerDied","Data":"b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102"}
Nov 01 20:16:12.947897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.944904    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg" event={"ID":"01c2fad9-a87d-47ca-b6b9-2df730d1d529","Type":"ContainerDied","Data":"85a8ddd416789df64ea6f99114c27d64f46aa42886e05ba0db4efc93568d9b08"}
Nov 01 20:16:12.947897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.944991    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"
Nov 01 20:16:12.961660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.953403    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m" podStartSLOduration=1.549179394 podStartE2EDuration="1.953389933s" totalImagesPullingTime="404.210539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:12.952742039 +0000 UTC m=+1018.368374513" watchObservedRunningTime="2026-11-01 20:16:12.953389933 +0000 UTC m=+1018.369022407"
Nov 01 20:16:12.984957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:12.980388    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:13.030762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.024939    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"]
Nov 01 20:16:13.084525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.084479    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-75c5wpdkg"]
Nov 01 20:16:13.084754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.084557    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"]
Nov 01 20:16:13.085222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.085198    6479 scope.go:122] "RemoveContainer" containerID="a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5"
Nov 01 20:16:13.094589 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:13.094542    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5\": container with ID starting with a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5 not found: ID does not exist" containerID="a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5"
Nov 01 20:16:13.094767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.094594    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5"} err="failed to get container status \"a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5\": rpc error: code = NotFound desc = could not find container \"a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5\": container with ID starting with a886bf8c6ceb8c9087f8fa18553f9a04102f79faf4160cbf5261c25e9133a0c5 not found: ID does not exist"
Nov 01 20:16:13.094767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.094618    6479 scope.go:122] "RemoveContainer" containerID="b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102"
Nov 01 20:16:13.095269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.095243    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:13.111909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.111253    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-7789nb87p"]
Nov 01 20:16:13.115111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.115088    6479 scope.go:122] "RemoveContainer" containerID="b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102"
Nov 01 20:16:13.115562 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:13.115538    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102\": container with ID starting with b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102 not found: ID does not exist" containerID="b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102"
Nov 01 20:16:13.115720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.115695    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102"} err="failed to get container status \"b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102\": rpc error: code = NotFound desc = could not find container \"b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102\": container with ID starting with b51521e10f83f4b8185283d9f1ec9674d43cdaaafa8968f116b5cf4206b7b102 not found: ID does not exist"
Nov 01 20:16:13.720910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.720869    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:16:13.875244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.875215    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:13.925246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:13.925092    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:14.020178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.020089    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:14.093680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.093630    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"]
Nov 01 20:16:14.104776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.104742    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"]
Nov 01 20:16:14.105487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.105466    6479 cpu_manager.go:414] "containerMap: removing container" podUID="01c2fad9-a87d-47ca-b6b9-2df730d1d529" containerName="payload-processing"
Nov 01 20:16:14.105487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.105487    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c2fad9-a87d-47ca-b6b9-2df730d1d529" containerName="payload-processing"
Nov 01 20:16:14.105733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.105542    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" containerName="payload-processing"
Nov 01 20:16:14.105733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.105551    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" containerName="payload-processing"
Nov 01 20:16:14.105837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.105749    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c2fad9-a87d-47ca-b6b9-2df730d1d529" containerName="payload-processing"
Nov 01 20:16:14.105837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.105792    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" containerName="payload-processing"
Nov 01 20:16:14.110944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.110908    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:14.111054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.110999    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.116380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.116355    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"]
Nov 01 20:16:14.122686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.119229    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"agc-cred-20ab74e39ebb\""
Nov 01 20:16:14.122686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.120675    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:14.195129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.195091    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.195338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.195218    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.195338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.195312    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.195474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.195357    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvsv\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.258364 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.258334    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:14.296513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.296439    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.296695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.296545    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.296695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.296585    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvsv\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.296811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.296770    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.297400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.297374    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.297564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.297539    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.299452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.299426    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.304890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.304870    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvsv\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.369667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.357322    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"]
Nov 01 20:16:14.382804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.382768    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"]
Nov 01 20:16:14.397667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.392479    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"]
Nov 01 20:16:14.397667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.392591    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.409675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.403241    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-90a5012dcf9c\""
Nov 01 20:16:14.410841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.410811    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:14.425343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.425287    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.427928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.427846    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.427928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.427897    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.428151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.428045    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6h6f\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.435237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.435211    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:14.529426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.529390    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.529586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.529432    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.529586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.529453    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.529586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.529565    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6h6f\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.530661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.530625    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.531102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.531081    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.533732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.533689    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.537260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.537232    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6h6f\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.708875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.708840    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c2fad9-a87d-47ca-b6b9-2df730d1d529" path="/var/lib/kubelet/pods/01c2fad9-a87d-47ca-b6b9-2df730d1d529/volumes"
Nov 01 20:16:14.709402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.709382    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e" path="/var/lib/kubelet/pods/9e5871aa-a1e0-4ad9-8c3e-08dd4edc946e/volumes"
Nov 01 20:16:14.735328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.735298    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"]
Nov 01 20:16:14.742949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.742927    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:14.747477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:14.747460    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:16:14.749099 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:14.749077    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ff5d09_a7c3_45ff_b07a_a81af4cc6ae9.slice/crio-74874367b6baf02c87c163b40bd0e4ba573b35c9cc85403e365f55d3a8bd420a WatchSource:0}: Error finding container 74874367b6baf02c87c163b40bd0e4ba573b35c9cc85403e365f55d3a8bd420a: Status 404 returned error can't find the container with id 74874367b6baf02c87c163b40bd0e4ba573b35c9cc85403e365f55d3a8bd420a
Nov 01 20:16:15.141871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.141840    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"]
Nov 01 20:16:15.145075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.145044    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7" event={"ID":"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9","Type":"ContainerStarted","Data":"77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0"}
Nov 01 20:16:15.145269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.145252    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:15.145429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.145414    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7" event={"ID":"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9","Type":"ContainerStarted","Data":"74874367b6baf02c87c163b40bd0e4ba573b35c9cc85403e365f55d3a8bd420a"}
Nov 01 20:16:15.159004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.158968    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:15.168798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.168745    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7" podStartSLOduration=1.168714028 podStartE2EDuration="1.168714028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:15.167204473 +0000 UTC m=+1020.582836947" watchObservedRunningTime="2026-11-01 20:16:15.168714028 +0000 UTC m=+1020.584346502"
Nov 01 20:16:15.859413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:15.859346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:16.092573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:16.092539    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:16:16.205726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:16.190836    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq" event={"ID":"53466176-a2d9-4641-9afb-95f586f051f5","Type":"ContainerStarted","Data":"1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc"}
Nov 01 20:16:16.205726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:16.190899    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq" event={"ID":"53466176-a2d9-4641-9afb-95f586f051f5","Type":"ContainerStarted","Data":"3e316f5e9f041919bcc022fa19907611235f9c515b8db1b42bc40660db0f4a58"}
Nov 01 20:16:16.205726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:16.190930    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:16.246670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:16.243374    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq" podStartSLOduration=2.243355904 podStartE2EDuration="2.243355904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:16.208074802 +0000 UTC m=+1021.623707276" watchObservedRunningTime="2026-11-01 20:16:16.243355904 +0000 UTC m=+1021.658988378"
Nov 01 20:16:16.561425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:16.561336    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:18.034383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:18.034350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:18.792626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:18.792593    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:16:18.792819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:18.792709    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:16:18.947791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:18.947757    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:19.122765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.103716    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz" podUID="d2fff25d-3468-48e2-b697-cc404a2c8c8f" containerName="payload-processing" containerID="cri-o://4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d" gracePeriod=25
Nov 01 20:16:19.366136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.363712    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck" podUID="711ea31a-e508-4ae9-abe7-28a4755776b0" containerName="payload-processing" containerID="cri-o://3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d" gracePeriod=25
Nov 01 20:16:19.492461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.492428    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:19.579724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.569531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:19.600350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.600309    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials\") pod \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") "
Nov 01 20:16:19.600551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.600416    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume\") pod \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") "
Nov 01 20:16:19.600551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.600458    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp\") pod \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") "
Nov 01 20:16:19.600551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.600493    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume\") pod \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\" (UID: \"d2fff25d-3468-48e2-b697-cc404a2c8c8f\") "
Nov 01 20:16:19.600822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.600790    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume" pod "d2fff25d-3468-48e2-b697-cc404a2c8c8f" (UID: "d2fff25d-3468-48e2-b697-cc404a2c8c8f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:19.600948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.600921    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume" pod "d2fff25d-3468-48e2-b697-cc404a2c8c8f" (UID: "d2fff25d-3468-48e2-b697-cc404a2c8c8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:19.601190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.601168    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.601244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.601199    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fff25d-3468-48e2-b697-cc404a2c8c8f-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.602585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.602552    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials" pod "d2fff25d-3468-48e2-b697-cc404a2c8c8f" (UID: "d2fff25d-3468-48e2-b697-cc404a2c8c8f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:19.602704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.602632    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp" pod "d2fff25d-3468-48e2-b697-cc404a2c8c8f" (UID: "d2fff25d-3468-48e2-b697-cc404a2c8c8f"). InnerVolumeSpecName "kube-api-access-8kwkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:19.744674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.744617    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.745116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.744978    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwkp\" (UniqueName: \"kubernetes.io/projected/d2fff25d-3468-48e2-b697-cc404a2c8c8f-kube-api-access-8kwkp\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.842205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.842176    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:19.846073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.846046    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v\") pod \"711ea31a-e508-4ae9-abe7-28a4755776b0\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") "
Nov 01 20:16:19.846220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.846129    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume\") pod \"711ea31a-e508-4ae9-abe7-28a4755776b0\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") "
Nov 01 20:16:19.846305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.846287    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume\") pod \"711ea31a-e508-4ae9-abe7-28a4755776b0\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") "
Nov 01 20:16:19.846366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.846338    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials\") pod \"711ea31a-e508-4ae9-abe7-28a4755776b0\" (UID: \"711ea31a-e508-4ae9-abe7-28a4755776b0\") "
Nov 01 20:16:19.846619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.846584    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume" pod "711ea31a-e508-4ae9-abe7-28a4755776b0" (UID: "711ea31a-e508-4ae9-abe7-28a4755776b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:19.846724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.846682    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume" pod "711ea31a-e508-4ae9-abe7-28a4755776b0" (UID: "711ea31a-e508-4ae9-abe7-28a4755776b0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:19.847076 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.847049    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.847190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.847081    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711ea31a-e508-4ae9-abe7-28a4755776b0-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.848336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.848303    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v" pod "711ea31a-e508-4ae9-abe7-28a4755776b0" (UID: "711ea31a-e508-4ae9-abe7-28a4755776b0"). InnerVolumeSpecName "kube-api-access-stx7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:19.848425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.848378    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials" pod "711ea31a-e508-4ae9-abe7-28a4755776b0" (UID: "711ea31a-e508-4ae9-abe7-28a4755776b0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:19.948125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.948086    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:19.948125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:19.948115    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stx7v\" (UniqueName: \"kubernetes.io/projected/711ea31a-e508-4ae9-abe7-28a4755776b0-kube-api-access-stx7v\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:20.151347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.151320    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:16:20.230578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.230477    6479 generic.go:356] "Generic (PLEG): container finished" podID="711ea31a-e508-4ae9-abe7-28a4755776b0" containerID="3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d" exitCode=0
Nov 01 20:16:20.232781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.231458    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck" event={"ID":"711ea31a-e508-4ae9-abe7-28a4755776b0","Type":"ContainerDied","Data":"3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d"}
Nov 01 20:16:20.232904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.231551    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"
Nov 01 20:16:20.237690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.234887    6479 generic.go:356] "Generic (PLEG): container finished" podID="d2fff25d-3468-48e2-b697-cc404a2c8c8f" containerID="4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d" exitCode=0
Nov 01 20:16:20.237690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.234993    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"
Nov 01 20:16:20.241736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.241055    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck" event={"ID":"711ea31a-e508-4ae9-abe7-28a4755776b0","Type":"ContainerDied","Data":"cfefafed2524a9c9e4bd1a5e67dccd4c4d0b3235c2464c13dd396735a202dd92"}
Nov 01 20:16:20.241736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.241202    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz" event={"ID":"d2fff25d-3468-48e2-b697-cc404a2c8c8f","Type":"ContainerDied","Data":"4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d"}
Nov 01 20:16:20.241736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.241219    6479 scope.go:122] "RemoveContainer" containerID="3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d"
Nov 01 20:16:20.241736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.241237    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz" event={"ID":"d2fff25d-3468-48e2-b697-cc404a2c8c8f","Type":"ContainerDied","Data":"80ae59141f0b8327268fec80159a95c040c169401cdca3cd5c0c6e5e862bfd84"}
Nov 01 20:16:20.289853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.289826    6479 scope.go:122] "RemoveContainer" containerID="3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d"
Nov 01 20:16:20.290215 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:20.290194    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d\": container with ID starting with 3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d not found: ID does not exist" containerID="3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d"
Nov 01 20:16:20.290304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.290223    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d"} err="failed to get container status \"3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d\": rpc error: code = NotFound desc = could not find container \"3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d\": container with ID starting with 3672b30521d441ae34f72e607a6c35e12f30a9c633c028d40c7590d87776199d not found: ID does not exist"
Nov 01 20:16:20.290304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.290245    6479 scope.go:122] "RemoveContainer" containerID="4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d"
Nov 01 20:16:20.294846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.294785    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"]
Nov 01 20:16:20.297910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.297890    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdkkpck"]
Nov 01 20:16:20.301559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.301541    6479 scope.go:122] "RemoveContainer" containerID="4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d"
Nov 01 20:16:20.301873 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:20.301855    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d\": container with ID starting with 4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d not found: ID does not exist" containerID="4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d"
Nov 01 20:16:20.301928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.301883    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d"} err="failed to get container status \"4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d\": rpc error: code = NotFound desc = could not find container \"4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d\": container with ID starting with 4e101293209bb583398db5a37413eab20d3dc99041658d428984ae41e4758a1d not found: ID does not exist"
Nov 01 20:16:20.308201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.308179    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"]
Nov 01 20:16:20.314003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.313984    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458zsllz"]
Nov 01 20:16:20.334553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.334531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:20.486281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.486239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:20.581547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.581497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:20.709994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.709959    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711ea31a-e508-4ae9-abe7-28a4755776b0" path="/var/lib/kubelet/pods/711ea31a-e508-4ae9-abe7-28a4755776b0/volumes"
Nov 01 20:16:20.710454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.710439    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fff25d-3468-48e2-b697-cc404a2c8c8f" path="/var/lib/kubelet/pods/d2fff25d-3468-48e2-b697-cc404a2c8c8f/volumes"
Nov 01 20:16:20.993671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:20.987673    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:21.156670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:21.141548    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:21.493842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:21.493808    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:21.585668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:21.584921    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:21.972677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:21.969005    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:22.078672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:22.075509    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:22.742960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:22.742925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:23.351710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:23.351672    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:24.151966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:24.151922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:24.671616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:24.671579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:24.949736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:24.949700    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:25.054616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:25.054579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:25.150992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:25.150952    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:25.261722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:25.260724    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:25.857991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:25.857954    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:26.129889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.129159    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:26.169791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.169747    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"]
Nov 01 20:16:26.170716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.170686    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d2fff25d-3468-48e2-b697-cc404a2c8c8f" containerName="payload-processing"
Nov 01 20:16:26.170882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.170721    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fff25d-3468-48e2-b697-cc404a2c8c8f" containerName="payload-processing"
Nov 01 20:16:26.170882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.170755    6479 cpu_manager.go:414] "containerMap: removing container" podUID="711ea31a-e508-4ae9-abe7-28a4755776b0" containerName="payload-processing"
Nov 01 20:16:26.170882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.170765    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711ea31a-e508-4ae9-abe7-28a4755776b0" containerName="payload-processing"
Nov 01 20:16:26.171052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.170957    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="711ea31a-e508-4ae9-abe7-28a4755776b0" containerName="payload-processing"
Nov 01 20:16:26.171052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.170988    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fff25d-3468-48e2-b697-cc404a2c8c8f" containerName="payload-processing"
Nov 01 20:16:26.177515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.177485    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.184677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.184635    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"]
Nov 01 20:16:26.185088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.184815    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-9be196\""
Nov 01 20:16:26.190841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.190405    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.194080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.194008    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"]
Nov 01 20:16:26.197604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.197525    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-9be196-dockercfg-5lmt9\""
Nov 01 20:16:26.199214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.198953    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:16:26.211661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.204885    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"]
Nov 01 20:16:26.244792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.244759    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:26.269809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.269401    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8tcj\" (UniqueName: \"kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj\") pod \"payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.269809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.269573    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.277538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.276251    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"]
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.447860    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.447906    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mrg\" (UniqueName: \"kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg\") pod \"payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.448213    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tcj\" (UniqueName: \"kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj\") pod \"payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.448442    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.449335    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.495359    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tcj\" (UniqueName: \"kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj\") pod \"payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.550731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.535923    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:26.567985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.552796    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.567985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.552846    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mrg\" (UniqueName: \"kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg\") pod \"payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.567985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.554467    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.567985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.566159    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mrg\" (UniqueName: \"kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg\") pod \"payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.759149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.759107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:26.860725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.849088    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:26.920390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:26.920352    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"]
Nov 01 20:16:26.926724 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:26.926538    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeede4389_f540_4c70_832e_a55e6198a442.slice/crio-1e2c9e804c904d420211ccd8e57cd1affaf6905d849e03140df3d094eb0daaba WatchSource:0}: Error finding container 1e2c9e804c904d420211ccd8e57cd1affaf6905d849e03140df3d094eb0daaba: Status 404 returned error can't find the container with id 1e2c9e804c904d420211ccd8e57cd1affaf6905d849e03140df3d094eb0daaba
Nov 01 20:16:27.201765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.201733    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"]
Nov 01 20:16:27.201903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.201842    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:16:27.244822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.244800    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"]
Nov 01 20:16:27.438289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.438259    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:27.559363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.548957    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:27.567057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.562741    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" event={"ID":"51356aec-ac1a-46a0-8eb2-cb56aa4e276a","Type":"ContainerStarted","Data":"3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c"}
Nov 01 20:16:27.568250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.568227    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:27.568572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.568548    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" event={"ID":"51356aec-ac1a-46a0-8eb2-cb56aa4e276a","Type":"ContainerStarted","Data":"c6ed8d2a2df3aad1c234a73cef449846bad8f71aa0db4cb6162ef407a1820f16"}
Nov 01 20:16:27.568722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.568707    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" event={"ID":"eede4389-f540-4c70-832e-a55e6198a442","Type":"ContainerStarted","Data":"f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771"}
Nov 01 20:16:27.568846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.568819    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:27.569004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.568977    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" event={"ID":"eede4389-f540-4c70-832e-a55e6198a442","Type":"ContainerStarted","Data":"1e2c9e804c904d420211ccd8e57cd1affaf6905d849e03140df3d094eb0daaba"}
Nov 01 20:16:27.600669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.598851    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" podStartSLOduration=1.59883065 podStartE2EDuration="1.59883065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:27.586312995 +0000 UTC m=+1033.001945469" watchObservedRunningTime="2026-11-01 20:16:27.59883065 +0000 UTC m=+1033.014463128"
Nov 01 20:16:27.611311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.611254    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" podStartSLOduration=1.6104137060000001 podStartE2EDuration="1.610413706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:27.608727867 +0000 UTC m=+1033.024360341" watchObservedRunningTime="2026-11-01 20:16:27.610413706 +0000 UTC m=+1033.026046181"
Nov 01 20:16:27.831345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.831265    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"]
Nov 01 20:16:27.841392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.841360    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"]
Nov 01 20:16:27.892879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.891737    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:27.991626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:27.991587    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:28.092089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.091971    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:28.145957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.145920    6479 patch_prober.go:29] interesting pod/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:16:28.249320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.146245    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:16:28.576534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.576501    6479 generic.go:356] "Generic (PLEG): container finished" podID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerID="191402052ef6d892fea7db67f6074ea7ad5a5866758ae2b393b151bf9722519a" exitCode=0
Nov 01 20:16:28.577011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.576557    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" event={"ID":"09fdd51b-5f92-4df4-8cc6-40eef12c0378","Type":"ContainerDied","Data":"191402052ef6d892fea7db67f6074ea7ad5a5866758ae2b393b151bf9722519a"}
Nov 01 20:16:28.577276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.577258    6479 scope.go:122] "RemoveContainer" containerID="191402052ef6d892fea7db67f6074ea7ad5a5866758ae2b393b151bf9722519a"
Nov 01 20:16:28.578663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.578620    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:28.578790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.578734    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:28.578860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.578788    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:28.743522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:28.743493    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:29.247543 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:29.247509    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:16:29.582573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:29.582469    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:29.592409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:29.592374    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" event={"ID":"09fdd51b-5f92-4df4-8cc6-40eef12c0378","Type":"ContainerStarted","Data":"f7ab69cb6ec6c97e02634f616f5570fb71a8da9b7c11e3acc94b7936bea95aa7"}
Nov 01 20:16:30.086249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.086214    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:16:30.086618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.086594    6479 patch_prober.go:29] interesting pod/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:16:30.086815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.086788    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 01 20:16:30.396924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.396837    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"]
Nov 01 20:16:30.397429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.397371    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f" podUID="f6004cd0-426b-4be9-bf90-2a1b7f4be179" containerName="maas-api" containerID="cri-o://ca500f39b681c054b689475337cdf89af7b725eb97d7f68af9ddc12f0a305a18" gracePeriod=30
Nov 01 20:16:30.429490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.429370    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:30.654723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.627124    6479 generic.go:356] "Generic (PLEG): container finished" podID="f6004cd0-426b-4be9-bf90-2a1b7f4be179" containerID="ca500f39b681c054b689475337cdf89af7b725eb97d7f68af9ddc12f0a305a18" exitCode=0
Nov 01 20:16:30.654723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:30.628041    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f" event={"ID":"f6004cd0-426b-4be9-bf90-2a1b7f4be179","Type":"ContainerDied","Data":"ca500f39b681c054b689475337cdf89af7b725eb97d7f68af9ddc12f0a305a18"}
Nov 01 20:16:31.094670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.084728    6479 patch_prober.go:29] interesting pod/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 01 20:16:31.094670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.084787    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 01 20:16:31.281672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.277564    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97" podUID="5b56a4f4-c30c-4957-9c90-041964cfd673" containerName="payload-processing" containerID="cri-o://eb0c127927dfbbe8411be8b9ca7ea5054443ec7d9ad1d996c0b64e1cbbccdcab" gracePeriod=25
Nov 01 20:16:31.298568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.298291    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:31.312508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.312480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:31.409888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.409811    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:31.487666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.484176    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn\" (UniqueName: \"kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn\") pod \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") "
Nov 01 20:16:31.487666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.484307    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca\") pod \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") "
Nov 01 20:16:31.487666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.484348    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs\") pod \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") "
Nov 01 20:16:31.487666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.484426    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\") pod \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\" (UID: \"f6004cd0-426b-4be9-bf90-2a1b7f4be179\") "
Nov 01 20:16:31.499685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.498091    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls" pod "f6004cd0-426b-4be9-bf90-2a1b7f4be179" (UID: "f6004cd0-426b-4be9-bf90-2a1b7f4be179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:16:31.499685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.498357    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn" pod "f6004cd0-426b-4be9-bf90-2a1b7f4be179" (UID: "f6004cd0-426b-4be9-bf90-2a1b7f4be179"). InnerVolumeSpecName "kube-api-access-lv2nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:31.507346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.502832    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca" pod "f6004cd0-426b-4be9-bf90-2a1b7f4be179" (UID: "f6004cd0-426b-4be9-bf90-2a1b7f4be179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:31.525679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.508227    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs" pod "f6004cd0-426b-4be9-bf90-2a1b7f4be179" (UID: "f6004cd0-426b-4be9-bf90-2a1b7f4be179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:16:31.594017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.593877    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv2nn\" (UniqueName: \"kubernetes.io/projected/f6004cd0-426b-4be9-bf90-2a1b7f4be179-kube-api-access-lv2nn\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:31.594017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.593920    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6004cd0-426b-4be9-bf90-2a1b7f4be179-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:31.594017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.593937    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:31.594017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.593951    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6004cd0-426b-4be9-bf90-2a1b7f4be179-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:31.608688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.599559    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"]
Nov 01 20:16:31.609449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.609373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:31.642831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.642779    6479 generic.go:356] "Generic (PLEG): container finished" podID="5b56a4f4-c30c-4957-9c90-041964cfd673" containerID="eb0c127927dfbbe8411be8b9ca7ea5054443ec7d9ad1d996c0b64e1cbbccdcab" exitCode=0
Nov 01 20:16:31.642973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.642851    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97" event={"ID":"5b56a4f4-c30c-4957-9c90-041964cfd673","Type":"ContainerDied","Data":"eb0c127927dfbbe8411be8b9ca7ea5054443ec7d9ad1d996c0b64e1cbbccdcab"}
Nov 01 20:16:31.644817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.644787    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy" containerID="cri-o://f7ab69cb6ec6c97e02634f616f5570fb71a8da9b7c11e3acc94b7936bea95aa7" gracePeriod=30
Nov 01 20:16:31.645163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.645141    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"
Nov 01 20:16:31.645747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.645718    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f" event={"ID":"f6004cd0-426b-4be9-bf90-2a1b7f4be179","Type":"ContainerDied","Data":"81a0348ca7b4279ff13a02ce941a920a24320afee693f0bc1330aef0bf77c019"}
Nov 01 20:16:31.645844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.645785    6479 scope.go:122] "RemoveContainer" containerID="ca500f39b681c054b689475337cdf89af7b725eb97d7f68af9ddc12f0a305a18"
Nov 01 20:16:31.991060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:31.984047    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"]
Nov 01 20:16:32.021461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.021421    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:32.036629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.034057    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea421713-696f87c995-jg92f"]
Nov 01 20:16:32.076311 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:32.076266    6479 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6004cd0_426b_4be9_bf90_2a1b7f4be179.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6004cd0_426b_4be9_bf90_2a1b7f4be179.slice/crio-81a0348ca7b4279ff13a02ce941a920a24320afee693f0bc1330aef0bf77c019\": RecentStats: unable to find data in memory cache]"
Nov 01 20:16:32.094295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.094263    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:16:32.132562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.132530    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h\") pod \"5b56a4f4-c30c-4957-9c90-041964cfd673\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") "
Nov 01 20:16:32.132759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.132606    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume\") pod \"5b56a4f4-c30c-4957-9c90-041964cfd673\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") "
Nov 01 20:16:32.132759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.132735    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials\") pod \"5b56a4f4-c30c-4957-9c90-041964cfd673\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") "
Nov 01 20:16:32.132886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.132764    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume\") pod \"5b56a4f4-c30c-4957-9c90-041964cfd673\" (UID: \"5b56a4f4-c30c-4957-9c90-041964cfd673\") "
Nov 01 20:16:32.133005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.132980    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume" pod "5b56a4f4-c30c-4957-9c90-041964cfd673" (UID: "5b56a4f4-c30c-4957-9c90-041964cfd673"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:32.133337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.133319    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.133428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.133364    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume" pod "5b56a4f4-c30c-4957-9c90-041964cfd673" (UID: "5b56a4f4-c30c-4957-9c90-041964cfd673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:32.134948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.134924    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h" pod "5b56a4f4-c30c-4957-9c90-041964cfd673" (UID: "5b56a4f4-c30c-4957-9c90-041964cfd673"). InnerVolumeSpecName "kube-api-access-77d8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:32.135738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.135715    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials" pod "5b56a4f4-c30c-4957-9c90-041964cfd673" (UID: "5b56a4f4-c30c-4957-9c90-041964cfd673"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:32.234271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.234236    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.234271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.234271    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b56a4f4-c30c-4957-9c90-041964cfd673-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.234495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.234286    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77d8h\" (UniqueName: \"kubernetes.io/projected/5b56a4f4-c30c-4957-9c90-041964cfd673-kube-api-access-77d8h\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.245504 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.245419    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf" podUID="6b2bb168-0fb3-4e3f-9fad-694b2ad81228" containerName="payload-processing" containerID="cri-o://097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1" gracePeriod=25
Nov 01 20:16:32.601682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.600369    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:16:32.640523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.640478    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume\") pod \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") "
Nov 01 20:16:32.640690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.640542    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume\") pod \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") "
Nov 01 20:16:32.640734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.640705    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials\") pod \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") "
Nov 01 20:16:32.640776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.640755    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm\") pod \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\" (UID: \"6b2bb168-0fb3-4e3f-9fad-694b2ad81228\") "
Nov 01 20:16:32.641857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.641827    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume" pod "6b2bb168-0fb3-4e3f-9fad-694b2ad81228" (UID: "6b2bb168-0fb3-4e3f-9fad-694b2ad81228"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:32.641998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.641910    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume" pod "6b2bb168-0fb3-4e3f-9fad-694b2ad81228" (UID: "6b2bb168-0fb3-4e3f-9fad-694b2ad81228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:32.643388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.643365    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm" pod "6b2bb168-0fb3-4e3f-9fad-694b2ad81228" (UID: "6b2bb168-0fb3-4e3f-9fad-694b2ad81228"). InnerVolumeSpecName "kube-api-access-85dnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:32.643613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.643595    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials" pod "6b2bb168-0fb3-4e3f-9fad-694b2ad81228" (UID: "6b2bb168-0fb3-4e3f-9fad-694b2ad81228"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:32.644151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.644127    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"]
Nov 01 20:16:32.645120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645102    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6b2bb168-0fb3-4e3f-9fad-694b2ad81228" containerName="payload-processing"
Nov 01 20:16:32.645168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645124    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2bb168-0fb3-4e3f-9fad-694b2ad81228" containerName="payload-processing"
Nov 01 20:16:32.645168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645142    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5b56a4f4-c30c-4957-9c90-041964cfd673" containerName="payload-processing"
Nov 01 20:16:32.645168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645150    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b56a4f4-c30c-4957-9c90-041964cfd673" containerName="payload-processing"
Nov 01 20:16:32.645258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645187    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f6004cd0-426b-4be9-bf90-2a1b7f4be179" containerName="maas-api"
Nov 01 20:16:32.645258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645196    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6004cd0-426b-4be9-bf90-2a1b7f4be179" containerName="maas-api"
Nov 01 20:16:32.645321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645315    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2bb168-0fb3-4e3f-9fad-694b2ad81228" containerName="payload-processing"
Nov 01 20:16:32.645357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645325    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b56a4f4-c30c-4957-9c90-041964cfd673" containerName="payload-processing"
Nov 01 20:16:32.645357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.645344    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6004cd0-426b-4be9-bf90-2a1b7f4be179" containerName="maas-api"
Nov 01 20:16:32.714103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.684759    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.714103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.689626    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-9be196-serving-cert\""
Nov 01 20:16:32.718545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.718516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:32.727916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.727864    6479 generic.go:356] "Generic (PLEG): container finished" podID="6b2bb168-0fb3-4e3f-9fad-694b2ad81228" containerID="097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1" exitCode=0
Nov 01 20:16:32.728147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.728066    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"
Nov 01 20:16:32.730317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.729877    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6004cd0-426b-4be9-bf90-2a1b7f4be179" path="/var/lib/kubelet/pods/f6004cd0-426b-4be9-bf90-2a1b7f4be179/volumes"
Nov 01 20:16:32.730624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.730607    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"]
Nov 01 20:16:32.730740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.730672    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf" event={"ID":"6b2bb168-0fb3-4e3f-9fad-694b2ad81228","Type":"ContainerDied","Data":"097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1"}
Nov 01 20:16:32.730740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.730714    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf" event={"ID":"6b2bb168-0fb3-4e3f-9fad-694b2ad81228","Type":"ContainerDied","Data":"603d1b971578bb5df40d6f30be26944d221418e7cb9b054acf2077f941e0c2ad"}
Nov 01 20:16:32.730856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.730744    6479 scope.go:122] "RemoveContainer" containerID="097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1"
Nov 01 20:16:32.732045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.731842    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"
Nov 01 20:16:32.732045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.731975    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97" event={"ID":"5b56a4f4-c30c-4957-9c90-041964cfd673","Type":"ContainerDied","Data":"50c090c8deef9aa7f52e2b44a187c542ae7a26b57d7a1adf9bd995d0e496ab17"}
Nov 01 20:16:32.742740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742712    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjl7\" (UniqueName: \"kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.742813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742778    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.742854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742822    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.742889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742877    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.742979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742966    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.743024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742981    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.743024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.742991    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.743024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.743000    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85dnm\" (UniqueName: \"kubernetes.io/projected/6b2bb168-0fb3-4e3f-9fad-694b2ad81228-kube-api-access-85dnm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:32.746221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.746199    6479 scope.go:122] "RemoveContainer" containerID="097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1"
Nov 01 20:16:32.746524 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:32.746503    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1\": container with ID starting with 097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1 not found: ID does not exist" containerID="097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1"
Nov 01 20:16:32.746601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.746531    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1"} err="failed to get container status \"097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1\": rpc error: code = NotFound desc = could not find container \"097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1\": container with ID starting with 097e341789fcb5e25e6ab57a2b7211b98581b7acd153de8167c0f01c105fe8f1 not found: ID does not exist"
Nov 01 20:16:32.746601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.746550    6479 scope.go:122] "RemoveContainer" containerID="eb0c127927dfbbe8411be8b9ca7ea5054443ec7d9ad1d996c0b64e1cbbccdcab"
Nov 01 20:16:32.791062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.790985    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"]
Nov 01 20:16:32.794512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.794492    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797v6k97"]
Nov 01 20:16:32.799479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.799452    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"]
Nov 01 20:16:32.801362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.801335    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:32.803389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.803368    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679dzmrnf"]
Nov 01 20:16:32.832189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.832154    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" podUID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" containerName="payload-processing" containerID="cri-o://d255956879cae64dde4df102c2cbd3e3bcc64dce0836eed88ed222d3ccb560ef" gracePeriod=25
Nov 01 20:16:32.842638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.842607    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr" podUID="7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" containerName="payload-pre-processing" containerID="cri-o://67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1" gracePeriod=25
Nov 01 20:16:32.844556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.844528    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.844667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.844618    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.844760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.844741    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.844902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.844877    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjl7\" (UniqueName: \"kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.846575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.846555    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.848525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.848507    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.848631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.848549    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.892160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.891346    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjl7\" (UniqueName: \"kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7\") pod \"maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:32.925686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:32.915368    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:33.033990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.033963    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:33.416287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.415207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:33.623670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.622904    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:16:33.645319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.645284    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume\") pod \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") "
Nov 01 20:16:33.648305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.648277    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l\" (UniqueName: \"kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l\") pod \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\" (UID: \"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a\") "
Nov 01 20:16:33.649068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.645875    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume" pod "7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" (UID: "7c65e6c6-e940-4c67-be3e-ba2111b8ab1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:33.707192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.707152    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l" pod "7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" (UID: "7c65e6c6-e940-4c67-be3e-ba2111b8ab1a"). InnerVolumeSpecName "kube-api-access-d257l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:33.726312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.726279    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"]
Nov 01 20:16:33.727708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.727516    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" containerName="payload-pre-processing"
Nov 01 20:16:33.727708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.727540    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" containerName="payload-pre-processing"
Nov 01 20:16:33.727870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.727766    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" containerName="payload-pre-processing"
Nov 01 20:16:33.736184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.736162    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.745450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.745419    6479 generic.go:356] "Generic (PLEG): container finished" podID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" containerID="d255956879cae64dde4df102c2cbd3e3bcc64dce0836eed88ed222d3ccb560ef" exitCode=0
Nov 01 20:16:33.745587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.745564    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" event={"ID":"f3cb43f7-16f1-476b-af8d-ba77b9ddf770","Type":"ContainerDied","Data":"d255956879cae64dde4df102c2cbd3e3bcc64dce0836eed88ed222d3ccb560ef"}
Nov 01 20:16:33.745693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.745612    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" event={"ID":"f3cb43f7-16f1-476b-af8d-ba77b9ddf770","Type":"ContainerDied","Data":"406f8fccd34c3604562e4d133a7503b5fd9f5cf08141065326868c954c0709cc"}
Nov 01 20:16:33.745693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.745632    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="406f8fccd34c3604562e4d133a7503b5fd9f5cf08141065326868c954c0709cc"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.749762    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.749813    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.749851    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.749880    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bxh\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.749960    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.750035    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.750063    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.750253    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.750318    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.750431    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d257l\" (UniqueName: \"kubernetes.io/projected/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-kube-api-access-d257l\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:33.750741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.750448    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:33.751403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.751380    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-aaea41-openshift-default-dockercfg-7lvtz\""
Nov 01 20:16:33.797030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.780087    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"]
Nov 01 20:16:33.800190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.800143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:33.807438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.804507    6479 generic.go:356] "Generic (PLEG): container finished" podID="7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" containerID="67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1" exitCode=0
Nov 01 20:16:33.807438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.804532    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr" event={"ID":"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a","Type":"ContainerDied","Data":"67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1"}
Nov 01 20:16:33.807438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.806873    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr" event={"ID":"7c65e6c6-e940-4c67-be3e-ba2111b8ab1a","Type":"ContainerDied","Data":"76ae025f7068e3cbd526a0adfd2a0d96d6f95e87ac7f6fc8de65c110741e26d8"}
Nov 01 20:16:33.807438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.806914    6479 scope.go:122] "RemoveContainer" containerID="67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1"
Nov 01 20:16:33.807438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.804592    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"
Nov 01 20:16:33.810398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.809204    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:16:33.831906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.831883    6479 scope.go:122] "RemoveContainer" containerID="67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1"
Nov 01 20:16:33.832341 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:33.832288    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1\": container with ID starting with 67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1 not found: ID does not exist" containerID="67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1"
Nov 01 20:16:33.832341 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.832319    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1"} err="failed to get container status \"67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1\": rpc error: code = NotFound desc = could not find container \"67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1\": container with ID starting with 67708930a312ae13d1d8465d5ae3db2e0de4b91894a658e2bd98afba5963c3c1 not found: ID does not exist"
Nov 01 20:16:33.846336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.846312    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"]
Nov 01 20:16:33.851089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851069    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea421713-8655f769f-g2lqr"]
Nov 01 20:16:33.851457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851436    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume\") pod \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") "
Nov 01 20:16:33.851586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851571    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7\" (UniqueName: \"kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7\") pod \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\" (UID: \"f3cb43f7-16f1-476b-af8d-ba77b9ddf770\") "
Nov 01 20:16:33.851829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851807    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.851913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851869    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.851976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851939    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.852032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.851975    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.852032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.852013    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.852133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.852041    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bxh\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.852194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.852133    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.852252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.852191    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.852252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.852222    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.882724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.880962    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume" pod "f3cb43f7-16f1-476b-af8d-ba77b9ddf770" (UID: "f3cb43f7-16f1-476b-af8d-ba77b9ddf770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:33.886832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.886779    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7" pod "f3cb43f7-16f1-476b-af8d-ba77b9ddf770" (UID: "f3cb43f7-16f1-476b-af8d-ba77b9ddf770"). InnerVolumeSpecName "kube-api-access-6dmk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:33.895679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.894034    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.897161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.896312    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.897161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.896375    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.897161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.896594    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.897161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.897106    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.899467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.899439    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.900435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.900410    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bxh\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.901300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.901275    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.901441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.901333    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:33.913957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.913929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:33.915572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.915553    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"]
Nov 01 20:16:33.935154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.931637    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:16:33.957586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.957045    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:33.957586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:33.957084    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmk7\" (UniqueName: \"kubernetes.io/projected/f3cb43f7-16f1-476b-af8d-ba77b9ddf770-kube-api-access-6dmk7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:34.131671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.127839    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:34.474678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.474049    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:34.596183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.595428    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:34.599716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.598812    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"]
Nov 01 20:16:34.602701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.602163    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:34.602701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.602251    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:34.602701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.602295    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:34.712956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.712926    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b56a4f4-c30c-4957-9c90-041964cfd673" path="/var/lib/kubelet/pods/5b56a4f4-c30c-4957-9c90-041964cfd673/volumes"
Nov 01 20:16:34.713355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.713341    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2bb168-0fb3-4e3f-9fad-694b2ad81228" path="/var/lib/kubelet/pods/6b2bb168-0fb3-4e3f-9fad-694b2ad81228/volumes"
Nov 01 20:16:34.713717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.713702    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c65e6c6-e940-4c67-be3e-ba2111b8ab1a" path="/var/lib/kubelet/pods/7c65e6c6-e940-4c67-be3e-ba2111b8ab1a/volumes"
Nov 01 20:16:34.713970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.713944    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq" podUID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.107:50052\" within 1s: context deadline exceeded"
Nov 01 20:16:34.714083 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:34.714064    6479 logging.go:55] [core] [Channel #1291 SubChannel #1292]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.107:50052", ServerName: "10.134.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.107:50052: operation was canceled"
Nov 01 20:16:34.818147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.817670    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" event={"ID":"52b2a924-7fde-427d-892d-cfaf07899030","Type":"ContainerStarted","Data":"309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"}
Nov 01 20:16:34.818147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.817753    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" event={"ID":"52b2a924-7fde-427d-892d-cfaf07899030","Type":"ContainerStarted","Data":"77fd51054a82ef6cf26b3d1d4f6b8560463548a8484d900c06ba704e733a1ccf"}
Nov 01 20:16:34.821365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.821331    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq" event={"ID":"27c6e266-02c6-4d4c-8b78-e4ba53a3751d","Type":"ContainerStarted","Data":"301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea"}
Nov 01 20:16:34.821365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.821360    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"
Nov 01 20:16:34.821587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.821380    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq" event={"ID":"27c6e266-02c6-4d4c-8b78-e4ba53a3751d","Type":"ContainerStarted","Data":"355eeff9448f8f555802725556cd108c553434f8ef5cb5d8d8f8615fbf1ef66d"}
Nov 01 20:16:34.821702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.821681    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:34.840347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.840300    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podStartSLOduration=1.840286994 podStartE2EDuration="1.840286994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:34.839729847 +0000 UTC m=+1040.255362321" watchObservedRunningTime="2026-11-01 20:16:34.840286994 +0000 UTC m=+1040.255919460"
Nov 01 20:16:34.888724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.888689    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"]
Nov 01 20:16:34.900697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.893082    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea421713-6f77d8496c-76sgq"]
Nov 01 20:16:34.903665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.903543    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq" podStartSLOduration=2.312266515 podStartE2EDuration="2.903513798s" totalImagesPullingTime="591.247283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:34.903304943 +0000 UTC m=+1040.318937413" watchObservedRunningTime="2026-11-01 20:16:34.903513798 +0000 UTC m=+1040.319146273"
Nov 01 20:16:34.996879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:34.995790    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:35.116723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:35.115737    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:35.136462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:35.131676    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:35.136462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:35.131831    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:16:35.136462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:35.131877    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:16:35.850085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:35.850035    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:36.133087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:36.133010    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:36.711293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:36.711254    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" path="/var/lib/kubelet/pods/f3cb43f7-16f1-476b-af8d-ba77b9ddf770/volumes"
Nov 01 20:16:36.718425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:36.718400    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:36.856677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:36.856265    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:36.871687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:36.867996    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:16:37.101888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:37.101807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:37.201238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:37.201204    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:37.566641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:37.566587    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:38.271025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.270985    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:38.359327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.359292    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"]
Nov 01 20:16:38.360237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.360215    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" containerName="payload-processing"
Nov 01 20:16:38.360237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.360235    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" containerName="payload-processing"
Nov 01 20:16:38.360430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.360376    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cb43f7-16f1-476b-af8d-ba77b9ddf770" containerName="payload-processing"
Nov 01 20:16:38.365930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.365906    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.371246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.371028    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-aaea41\""
Nov 01 20:16:38.371804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.371779    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-aaea41-dockercfg-g65nm\""
Nov 01 20:16:38.375105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.374133    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.375105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.374258    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwzpv\" (UniqueName: \"kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv\") pod \"payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.382185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.382161    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"]
Nov 01 20:16:38.402503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.402470    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"]
Nov 01 20:16:38.406886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.406863    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.415558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.415527    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"]
Nov 01 20:16:38.487313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.487272    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg9c\" (UniqueName: \"kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c\") pod \"payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.487765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.487737    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.493669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.488035    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.493980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.493955    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwzpv\" (UniqueName: \"kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv\") pod \"payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.514668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.495247    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.609824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.609729    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg9c\" (UniqueName: \"kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c\") pod \"payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.647381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.647338    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.649937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.649909    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.674682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.674448    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwzpv\" (UniqueName: \"kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv\") pod \"payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.678669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.675439    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg9c\" (UniqueName: \"kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c\") pod \"payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.695225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.694955    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:38.732785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.732753    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:38.977490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:38.977463    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"]
Nov 01 20:16:38.978763 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:38.978738    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8f350c_b464_493d_8a6c_ef9bd7a9daa1.slice/crio-11febde413d26205fad70231d38188b7ca17493582777a0ff2e832384d475600 WatchSource:0}: Error finding container 11febde413d26205fad70231d38188b7ca17493582777a0ff2e832384d475600: Status 404 returned error can't find the container with id 11febde413d26205fad70231d38188b7ca17493582777a0ff2e832384d475600
Nov 01 20:16:39.036440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.036417    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"]
Nov 01 20:16:39.037965 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:39.037930    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacf3d06_22d0_4637_bc28_991f58a055ad.slice/crio-319a3eca7b5bfb7950b61b4a08d624136b8a46db1f193d4a136f58be12162787 WatchSource:0}: Error finding container 319a3eca7b5bfb7950b61b4a08d624136b8a46db1f193d4a136f58be12162787: Status 404 returned error can't find the container with id 319a3eca7b5bfb7950b61b4a08d624136b8a46db1f193d4a136f58be12162787
Nov 01 20:16:39.546739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.546703    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"]
Nov 01 20:16:39.550963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.550932    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.559926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.559891    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bccce1-openshift-default-dockercfg-5zk62\""
Nov 01 20:16:39.561202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.561181    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"]
Nov 01 20:16:39.596119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.596086    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:16:39.596319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.596175    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:16:39.664138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664095    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664206    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664266    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664395    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664545    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664580    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664633    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664713    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.664921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.664746    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqdr\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.697187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.697134    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.767772    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.770748    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.770795    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.770931    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.771041    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.771071    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.771123    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.771190    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.783676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.771219    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqdr\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.794360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.794325    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.797067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.794747    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.797256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.796226    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.797356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.796981    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.798328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.798283    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.800518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.800471    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.804157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.803821    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.811691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.804764    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqdr\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.833669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.833212    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.867273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.867238    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:39.884038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.884003    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" event={"ID":"bacf3d06-22d0-4637-bc28-991f58a055ad","Type":"ContainerStarted","Data":"8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5"}
Nov 01 20:16:39.884158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.884081    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:39.884158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.884108    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" event={"ID":"bacf3d06-22d0-4637-bc28-991f58a055ad","Type":"ContainerStarted","Data":"319a3eca7b5bfb7950b61b4a08d624136b8a46db1f193d4a136f58be12162787"}
Nov 01 20:16:39.885904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.885881    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" event={"ID":"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1","Type":"ContainerStarted","Data":"5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4"}
Nov 01 20:16:39.885982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.885926    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" event={"ID":"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1","Type":"ContainerStarted","Data":"11febde413d26205fad70231d38188b7ca17493582777a0ff2e832384d475600"}
Nov 01 20:16:39.886482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.886466    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:39.902008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:39.901956    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" podStartSLOduration=1.901941553 podStartE2EDuration="1.901941553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:39.901882404 +0000 UTC m=+1045.317514872" watchObservedRunningTime="2026-11-01 20:16:39.901941553 +0000 UTC m=+1045.317574027"
Nov 01 20:16:40.148844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.148791    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" podStartSLOduration=2.148771705 podStartE2EDuration="2.148771705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:39.92382692 +0000 UTC m=+1045.339459397" watchObservedRunningTime="2026-11-01 20:16:40.148771705 +0000 UTC m=+1045.564404178"
Nov 01 20:16:40.151199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.151132    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"]
Nov 01 20:16:40.151404 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:40.151375    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff981dbc_864a_40a9_8908_df562cae4c22.slice/crio-5b60a098bd6d96757ce0685bb6d6b892eee043fae761ab19c31924442c38c5eb WatchSource:0}: Error finding container 5b60a098bd6d96757ce0685bb6d6b892eee043fae761ab19c31924442c38c5eb: Status 404 returned error can't find the container with id 5b60a098bd6d96757ce0685bb6d6b892eee043fae761ab19c31924442c38c5eb
Nov 01 20:16:40.154276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.154246    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:40.154348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.154306    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:40.154348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.154332    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:16:40.275860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.275827    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:40.833418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.833391    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:16:40.897821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.897266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:40.929799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.929759    6479 generic.go:356] "Generic (PLEG): container finished" podID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerID="f7ab69cb6ec6c97e02634f616f5570fb71a8da9b7c11e3acc94b7936bea95aa7" exitCode=0
Nov 01 20:16:40.929966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.929911    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" event={"ID":"09fdd51b-5f92-4df4-8cc6-40eef12c0378","Type":"ContainerDied","Data":"f7ab69cb6ec6c97e02634f616f5570fb71a8da9b7c11e3acc94b7936bea95aa7"}
Nov 01 20:16:40.930056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.930001    6479 scope.go:122] "RemoveContainer" containerID="191402052ef6d892fea7db67f6074ea7ad5a5866758ae2b393b151bf9722519a"
Nov 01 20:16:40.933846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.933813    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" event={"ID":"ff981dbc-864a-40a9-8908-df562cae4c22","Type":"ContainerStarted","Data":"deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee"}
Nov 01 20:16:40.934059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:40.934041    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" event={"ID":"ff981dbc-864a-40a9-8908-df562cae4c22","Type":"ContainerStarted","Data":"5b60a098bd6d96757ce0685bb6d6b892eee043fae761ab19c31924442c38c5eb"}
Nov 01 20:16:41.297436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.297406    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:16:41.320555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.320483    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podStartSLOduration=2.320469864 podStartE2EDuration="2.320469864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:41.044381011 +0000 UTC m=+1046.460013484" watchObservedRunningTime="2026-11-01 20:16:41.320469864 +0000 UTC m=+1046.736102336"
Nov 01 20:16:41.418779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.418746    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.418963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.418812    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.418963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.418866    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.418963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.418891    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.419163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419015    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.419163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419038    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.419163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419057    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.419163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419095    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.419163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419142    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data\") pod \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\" (UID: \"09fdd51b-5f92-4df4-8cc6-40eef12c0378\") "
Nov 01 20:16:41.419952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419918    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:41.420060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.419927    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:41.420168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.420147    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:41.420224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.420157    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:41.420444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.420422    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:41.421943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.421910    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:41.422790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.422768    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:16:41.422882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.422803    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:41.422944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.422882    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x" pod "09fdd51b-5f92-4df4-8cc6-40eef12c0378" (UID: "09fdd51b-5f92-4df4-8cc6-40eef12c0378"). InnerVolumeSpecName "kube-api-access-qrz4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.522938    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.522979    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.522994    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.523007    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.523020    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.523035    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.523047    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.523061    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrz4x\" (UniqueName: \"kubernetes.io/projected/09fdd51b-5f92-4df4-8cc6-40eef12c0378-kube-api-access-qrz4x\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.539158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.523075    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fdd51b-5f92-4df4-8cc6-40eef12c0378-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:41.746245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.746166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:41.877715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.875414    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:41.877715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.875595    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:16:41.877715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.875655    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 01 20:16:41.996677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.986501    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"
Nov 01 20:16:41.998060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.998022    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs" event={"ID":"09fdd51b-5f92-4df4-8cc6-40eef12c0378","Type":"ContainerDied","Data":"90c8dcd6709c9495591b2c8e025725f68e686888d616fb8a9fc22a77870f48cc"}
Nov 01 20:16:41.998435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:41.998419    6479 scope.go:122] "RemoveContainer" containerID="f7ab69cb6ec6c97e02634f616f5570fb71a8da9b7c11e3acc94b7936bea95aa7"
Nov 01 20:16:42.034529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.032766    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"]
Nov 01 20:16:42.035693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.035667    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ea421713-openshift-default-7f4cbfcf9c-xjvbs"]
Nov 01 20:16:42.542351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.542318    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:42.719278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.719244    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" path="/var/lib/kubelet/pods/09fdd51b-5f92-4df4-8cc6-40eef12c0378/volumes"
Nov 01 20:16:42.868728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.868142    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:16:42.868728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.868202    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 01 20:16:42.927513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.927471    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"]
Nov 01 20:16:42.928518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.928488    6479 cpu_manager.go:414] "containerMap: removing container" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy"
Nov 01 20:16:42.928518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.928517    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy"
Nov 01 20:16:42.928734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.928564    6479 cpu_manager.go:414] "containerMap: removing container" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy"
Nov 01 20:16:42.928734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.928572    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy"
Nov 01 20:16:42.928855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.928789    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy"
Nov 01 20:16:42.928855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.928816    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fdd51b-5f92-4df4-8cc6-40eef12c0378" containerName="istio-proxy"
Nov 01 20:16:42.936974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.934730    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.939257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.939229    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"]
Nov 01 20:16:42.947519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.947483    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"kube-root-ca.crt\""
Nov 01 20:16:42.947952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.947932    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"openshift-service-ca.crt\""
Nov 01 20:16:42.948195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.948179    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"premium-w3-9de2ac-kserve-self-signed-certs\""
Nov 01 20:16:42.949730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.949711    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"default-dockercfg-8w4sw\""
Nov 01 20:16:42.990876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.990840    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.991282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.991244    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fbm\" (UniqueName: \"kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.991548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.991524    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.991794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.991774    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.991911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.991847    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.991911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.991890    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:42.992037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:42.991600    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:43.095673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.093500    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.095673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.093594    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.095673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.093633    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.095673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.093693    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.095673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.093838    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.095673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.093897    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fbm\" (UniqueName: \"kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.121709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.116170    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.121709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.116457    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.121709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.118412    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.121709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.121214    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.138669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.131378    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.138669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.137512    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fbm\" (UniqueName: \"kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm\") pod \"premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") " pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.251997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.251949    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:43.312515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.312481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:43.772753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.752623    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"]
Nov 01 20:16:43.871145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.871104    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:16:43.871269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.871169    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 01 20:16:43.944403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.944369    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"]
Nov 01 20:16:43.948802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:43.948779    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.003700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.002138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:44.004777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.004749    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-aaea41-serving-cert\""
Nov 01 20:16:44.010862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.010833    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"]
Nov 01 20:16:44.012940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.012892    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" event={"ID":"b3dce113-c837-4cd8-bcb9-0a8f41476159","Type":"ContainerStarted","Data":"81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0"}
Nov 01 20:16:44.012940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.012928    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" event={"ID":"b3dce113-c837-4cd8-bcb9-0a8f41476159","Type":"ContainerStarted","Data":"5990ed54aed69ea24c13cd6d8fdaf59a28884a612f8048f9ddac0d8d77656ea9"}
Nov 01 20:16:44.036772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.036674    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.036772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.036747    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.036998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.036779    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kk7\" (UniqueName: \"kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.036998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.036809    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.039087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.039039    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" podStartSLOduration=2.039026171 podStartE2EDuration="2.039026171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:44.037731118 +0000 UTC m=+1049.453363590" watchObservedRunningTime="2026-11-01 20:16:44.039026171 +0000 UTC m=+1049.454658662"
Nov 01 20:16:44.088302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.088263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:44.138385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.138349    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.138546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.138417    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.138546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.138446    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kk7\" (UniqueName: \"kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.138546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.138475    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.139189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.139162    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.140949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.140926    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.141062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.140973    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.145834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.145810    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kk7\" (UniqueName: \"kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7\") pod \"maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.190667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.190038    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:44.257684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.256902    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:44.307269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.305911    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:44.819072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.819050    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"]
Nov 01 20:16:44.877693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.877055    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:16:44.877693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.877117    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 01 20:16:44.884069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.883532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:44.985272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:44.985241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:45.081272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:45.081180    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl" event={"ID":"9cc72527-0c94-42e7-a7fb-a62b7bfc2440","Type":"ContainerStarted","Data":"d3a1dd058f90cad029c0a705f0e0b9e75b4c5795b2cdf19e934c8cbe4dcc9a6f"}
Nov 01 20:16:45.425835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:45.425810    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:45.667716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:45.664048    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:45.803462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:45.803406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:45.875525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:45.875057    6479 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:16:45.875525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:45.875115    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 01 20:16:46.104709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:46.104670    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl" event={"ID":"9cc72527-0c94-42e7-a7fb-a62b7bfc2440","Type":"ContainerStarted","Data":"c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68"}
Nov 01 20:16:46.119818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:46.119792    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:46.210120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:46.210054    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:46.510940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:46.510899    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:46.656837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:46.656787    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:47.067800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:47.067768    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:47.068979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:47.068954    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:47.069983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:47.069960    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:16:47.089157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:47.089110    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl" podStartSLOduration=3.5944622 podStartE2EDuration="4.089096372s" totalImagesPullingTime="494.634172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:46.146862737 +0000 UTC m=+1051.562495208" watchObservedRunningTime="2026-11-01 20:16:47.089096372 +0000 UTC m=+1052.504728871"
Nov 01 20:16:47.365924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:47.365813    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:47.630243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:47.630169    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:16:48.467539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:48.467498    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:49.478215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:49.478186    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:49.834999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:49.834922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:50.092178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:50.092099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:50.758212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:50.758179    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:50.937220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:50.937190    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:16:51.123705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.123592    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:51.197369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.197331    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"]
Nov 01 20:16:51.203249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.203215    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.204404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.204373    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"]
Nov 01 20:16:51.207809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.207781    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e83b4b-dockercfg-xntsc\""
Nov 01 20:16:51.208127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.208101    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e83b4b\""
Nov 01 20:16:51.235206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.235176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:51.255229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.255198    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"]
Nov 01 20:16:51.262137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.262113    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.270314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.270096    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.270314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.270200    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rn8g\" (UniqueName: \"kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g\") pod \"payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.276674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.275009    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"]
Nov 01 20:16:51.411750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.411286    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.411750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.411373    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.411750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.411443    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rn8g\" (UniqueName: \"kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g\") pod \"payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.411750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.411630    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctp7\" (UniqueName: \"kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7\") pod \"payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.423665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.419865    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.526369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.512759    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctp7\" (UniqueName: \"kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7\") pod \"payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.526369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.513197    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.526369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.523871    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.526369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.525695    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rn8g\" (UniqueName: \"kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g\") pod \"payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.537064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.535895    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:51.548715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.545565    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctp7\" (UniqueName: \"kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7\") pod \"payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.813432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.813394    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:51.855480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.855445    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"]
Nov 01 20:16:51.857855 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:51.857826    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a98fd7b_5575_4f8b_8a1e_4dda94d8c7da.slice/crio-8d1fd364fc84026b1a6f1e1589fa93d7f5ba5f5b2f6743ce4c1a3909de028a88 WatchSource:0}: Error finding container 8d1fd364fc84026b1a6f1e1589fa93d7f5ba5f5b2f6743ce4c1a3909de028a88: Status 404 returned error can't find the container with id 8d1fd364fc84026b1a6f1e1589fa93d7f5ba5f5b2f6743ce4c1a3909de028a88
Nov 01 20:16:51.874401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:51.874165    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:52.037776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.036588    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:16:52.111788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.111741    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"]
Nov 01 20:16:52.112855 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:16:52.112822    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8431cbc4_bd80_4332_a51d_155e91edb9da.slice/crio-19484c2f71ac4cae5152653fbf81e0f197ecf5d51592de59767b0f64e5e7d3b8 WatchSource:0}: Error finding container 19484c2f71ac4cae5152653fbf81e0f197ecf5d51592de59767b0f64e5e7d3b8: Status 404 returned error can't find the container with id 19484c2f71ac4cae5152653fbf81e0f197ecf5d51592de59767b0f64e5e7d3b8
Nov 01 20:16:52.153056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.153029    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:16:52.250480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.250441    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" event={"ID":"8431cbc4-bd80-4332-a51d-155e91edb9da","Type":"ContainerStarted","Data":"914d8227cc24e471de9b2c2e6e1b0bf48a891aac83435950d9cb4c9021fbc6cf"}
Nov 01 20:16:52.250722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.250511    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" event={"ID":"8431cbc4-bd80-4332-a51d-155e91edb9da","Type":"ContainerStarted","Data":"19484c2f71ac4cae5152653fbf81e0f197ecf5d51592de59767b0f64e5e7d3b8"}
Nov 01 20:16:52.252531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.252496    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" event={"ID":"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da","Type":"ContainerStarted","Data":"a4ce211bc6631bfec98c8875d883c2644fe70ad2ef5f7e1c83220b075297fd49"}
Nov 01 20:16:52.252710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.252554    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" event={"ID":"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da","Type":"ContainerStarted","Data":"8d1fd364fc84026b1a6f1e1589fa93d7f5ba5f5b2f6743ce4c1a3909de028a88"}
Nov 01 20:16:52.252777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.252743    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:16:52.268253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.268202    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" podStartSLOduration=1.268188544 podStartE2EDuration="1.268188544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:52.267529417 +0000 UTC m=+1057.683161891" watchObservedRunningTime="2026-11-01 20:16:52.268188544 +0000 UTC m=+1057.683821018"
Nov 01 20:16:52.752314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:52.752257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:53.255045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:53.254996    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:53.262130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:53.262093    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:16:53.286402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:53.286374    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:53.289225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:53.289164    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" podStartSLOduration=2.289145255 podStartE2EDuration="2.289145255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:53.283753159 +0000 UTC m=+1058.699385633" watchObservedRunningTime="2026-11-01 20:16:53.289145255 +0000 UTC m=+1058.704777728"
Nov 01 20:16:53.482466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:53.482253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:54.242144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:54.242110    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:54.272515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:54.272483    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:16:55.208931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:55.208899    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:56.036331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:56.036282    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:56.757126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:56.757050    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:57.195569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:57.195530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:57.549058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:57.548977    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:58.367403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:58.367352    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:58.470662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:58.470612    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:58.918184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:58.918133    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:59.246161 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:16:59.246113    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:16:59.252184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:59.251518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:59.347403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:59.347369    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:16:59.447552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:16:59.447518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:00.166443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:00.166405    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:01.006954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:01.006920    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:01.119729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:01.119692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:01.685906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:01.685863    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:02.998145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:02.998101    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:03.111971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:03.111920    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:03.218677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:03.218458    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:03.297770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:03.294843    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:17:04.300020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.299978    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:17:04.308706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.300425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:04.402695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.402637    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"]
Nov 01 20:17:04.407557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.407522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:04.411677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.411152    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"]
Nov 01 20:17:04.412793 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.412765    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.414324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.414299    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"]
Nov 01 20:17:04.423898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.421074    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6f810f-dockercfg-wz2b2\""
Nov 01 20:17:04.423898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.421217    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6f810f\""
Nov 01 20:17:04.423898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.421241    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:04.423898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.422524    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"]
Nov 01 20:17:04.668949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.668860    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:04.669296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.669269    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.669544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.669525    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsh9l\" (UniqueName: \"kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l\") pod \"payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.669728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.669706    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zdv\" (UniqueName: \"kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv\") pod \"payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:04.806995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.806959    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsh9l\" (UniqueName: \"kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l\") pod \"payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.811033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.810999    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zdv\" (UniqueName: \"kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv\") pod \"payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:04.811236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.811220    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:04.812462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.812439    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.813588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.812138    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:04.813755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.813373    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.816218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.816196    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsh9l\" (UniqueName: \"kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l\") pod \"payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:04.818177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:04.818129    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zdv\" (UniqueName: \"kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv\") pod \"payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:05.039515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.039476    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:05.076692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.076661    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:05.144206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.143386    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:05.593692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.592858    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"]
Nov 01 20:17:05.594157 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:17:05.593968    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbe6911_ff8e_48a4_80be_00a9b5da73f9.slice/crio-c92a6769b5846231860163a8adab00befacd804ca93fabea5b12ed4c96c59aa6 WatchSource:0}: Error finding container c92a6769b5846231860163a8adab00befacd804ca93fabea5b12ed4c96c59aa6: Status 404 returned error can't find the container with id c92a6769b5846231860163a8adab00befacd804ca93fabea5b12ed4c96c59aa6
Nov 01 20:17:05.597594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.596922    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"]
Nov 01 20:17:05.598539 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:17:05.598515    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745458a2_aff2_4647_9933_ded4c77f8832.slice/crio-f1b740d5ab2f60f8ceeea4aed80e9ad1de2695e97e79f84140abfad941fed891 WatchSource:0}: Error finding container f1b740d5ab2f60f8ceeea4aed80e9ad1de2695e97e79f84140abfad941fed891: Status 404 returned error can't find the container with id f1b740d5ab2f60f8ceeea4aed80e9ad1de2695e97e79f84140abfad941fed891
Nov 01 20:17:05.695408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.695211    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs" event={"ID":"ffbe6911-ff8e-48a4-80be-00a9b5da73f9","Type":"ContainerStarted","Data":"c92a6769b5846231860163a8adab00befacd804ca93fabea5b12ed4c96c59aa6"}
Nov 01 20:17:05.697296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.697267    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv" event={"ID":"745458a2-aff2-4647-9933-ded4c77f8832","Type":"ContainerStarted","Data":"f1b740d5ab2f60f8ceeea4aed80e9ad1de2695e97e79f84140abfad941fed891"}
Nov 01 20:17:05.746803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:05.746766    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:06.481677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.479145    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:06.721600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.721564    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv" event={"ID":"745458a2-aff2-4647-9933-ded4c77f8832","Type":"ContainerStarted","Data":"9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446"}
Nov 01 20:17:06.722021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.721633    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:06.722703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.722625    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs" event={"ID":"ffbe6911-ff8e-48a4-80be-00a9b5da73f9","Type":"ContainerStarted","Data":"59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82"}
Nov 01 20:17:06.722831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.722746    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:06.736466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.736375    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv" podStartSLOduration=2.736359277 podStartE2EDuration="2.736359277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:06.735596423 +0000 UTC m=+1072.151228897" watchObservedRunningTime="2026-11-01 20:17:06.736359277 +0000 UTC m=+1072.151991750"
Nov 01 20:17:06.755707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:06.752532    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs" podStartSLOduration=2.752516829 podStartE2EDuration="2.752516829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:06.750473384 +0000 UTC m=+1072.166105869" watchObservedRunningTime="2026-11-01 20:17:06.752516829 +0000 UTC m=+1072.168149302"
Nov 01 20:17:07.068303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:07.068214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:07.247069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:07.247030    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:07.775144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:07.775110    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:08.334445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:08.334406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:08.941738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:08.941702    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:09.851815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:09.851775    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:10.660963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:10.660928    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:11.269478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:11.269444    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:11.909996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:11.909839    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"]
Nov 01 20:17:11.916335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:11.916298    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:11.925199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:11.924498    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6f810f-serving-cert\""
Nov 01 20:17:11.931310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:11.931276    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"]
Nov 01 20:17:11.954719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:11.954607    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:12.030297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.030263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:12.033690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.033663    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.033843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.033783    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.033910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.033891    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.034009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.033995    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fnf\" (UniqueName: \"kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.136134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.136088    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.137470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.137430    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.138150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.138121    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.149481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.149097    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.149999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.149974    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fnf\" (UniqueName: \"kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.172584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.151812    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:12.194666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.186804    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.205244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.205212    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fnf\" (UniqueName: \"kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.240972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.240868    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs\") pod \"maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.273673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.270994    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:12.608682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.607159    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:12.625553 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:17:12.625492    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3953f2b_58ea_4e45_9308_9dedfba106c2.slice/crio-cb05e333c74d4b32507a8efe0e6710fc325c263e8ba1f7913e393b5b5d68c2fe WatchSource:0}: Error finding container cb05e333c74d4b32507a8efe0e6710fc325c263e8ba1f7913e393b5b5d68c2fe: Status 404 returned error can't find the container with id cb05e333c74d4b32507a8efe0e6710fc325c263e8ba1f7913e393b5b5d68c2fe
Nov 01 20:17:12.630454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.629735    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"]
Nov 01 20:17:12.848721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:12.848681    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8" event={"ID":"c3953f2b-58ea-4e45-9308-9dedfba106c2","Type":"ContainerStarted","Data":"cb05e333c74d4b32507a8efe0e6710fc325c263e8ba1f7913e393b5b5d68c2fe"}
Nov 01 20:17:13.338521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.338429    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:13.868081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.868042    6479 generic.go:356] "Generic (PLEG): container finished" podID="52b2a924-7fde-427d-892d-cfaf07899030" containerID="309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968" exitCode=0
Nov 01 20:17:13.868307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.868265    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" event={"ID":"52b2a924-7fde-427d-892d-cfaf07899030","Type":"ContainerDied","Data":"309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"}
Nov 01 20:17:13.868898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.868876    6479 scope.go:122] "RemoveContainer" containerID="309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"
Nov 01 20:17:13.871689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.871625    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:17:13.871829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.871796    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:17:13.871954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.871931    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:17:13.872990 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.872963    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8" event={"ID":"c3953f2b-58ea-4e45-9308-9dedfba106c2","Type":"ContainerStarted","Data":"2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45"}
Nov 01 20:17:13.873155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.873139    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:13.905471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:13.905407    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8" podStartSLOduration=2.528888895 podStartE2EDuration="2.905387262s" totalImagesPullingTime="376.498367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:13.903574108 +0000 UTC m=+1079.319206582" watchObservedRunningTime="2026-11-01 20:17:13.905387262 +0000 UTC m=+1079.321019736"
Nov 01 20:17:14.028605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:14.028520    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:14.670669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:14.668128    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:14.895765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:14.895731    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" event={"ID":"52b2a924-7fde-427d-892d-cfaf07899030","Type":"ContainerStarted","Data":"f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a"}
Nov 01 20:17:14.968710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:14.968608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:15.128808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:15.128768    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:17:15.128978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:15.128922    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:17:15.128978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:15.128965    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:17:15.275331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:15.275231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:15.894462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:15.894419    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:16.129131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:16.129094    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:17:16.129334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:16.129152    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:17:16.552845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:16.552810    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:16.979871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:16.979837    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:17.142433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:17.142400    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:17:17.142729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:17.142698    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:17:17.198761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:17.198725    6479 scope.go:122] "RemoveContainer" containerID="b3fb7da18ea3d5cd4af7ea851fb03d6fefad25b0c9bbb9d84c98b050a37127b4"
Nov 01 20:17:17.298118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:17.298034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:17.734173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:17.734136    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:17:17.735544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:17.735510    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:17:18.025309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:18.025273    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:18.134672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:18.132177    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:17:18.134672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:18.132269    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:17:18.674666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:18.674493    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:19.128780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:19.128744    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:17:19.129147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:19.128811    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:17:19.390289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:19.390191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:19.901394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:19.901363    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:17:20.128779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:20.128743    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 20:17:20.129267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:20.128812    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 20:17:20.161933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:20.161836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:20.949768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:20.949738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:21.132679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:21.132634    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:17:21.133124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:21.133105    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:17:21.133717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:21.133697    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:17:21.868593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:21.868549    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:17:22.019180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:22.019144    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:22.145738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:22.143543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:22.891534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:22.891488    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:23.314929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:23.314897    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:24.028499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:24.028459    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:24.127580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:24.127543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:24.304463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:24.304370    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:24.411159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:24.411123    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:24.510445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:24.510398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:24.845819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:24.845782    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:25.255023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:25.254985    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:25.356357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:25.356302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:25.475942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:25.475906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:25.970273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:25.970242    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:26.098920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.098884    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"]
Nov 01 20:17:26.104195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.104160    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.115221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.115079    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\""
Nov 01 20:17:26.116176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.116152    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\""
Nov 01 20:17:26.116792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.116773    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\""
Nov 01 20:17:26.117389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.117369    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\""
Nov 01 20:17:26.117819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.117793    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\""
Nov 01 20:17:26.118259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.118241    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-b2b285556e64\""
Nov 01 20:17:26.137251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.137216    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"]
Nov 01 20:17:26.391319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.382532    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52f2\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.417669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.398973    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.417669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.399331    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.417669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.399389    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.500781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.500737    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.501356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.500818    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.501356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.500878    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52f2\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.501356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.500929    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.501946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.501916    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.502081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.502028    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.503833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.503802    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.509179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.509156    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52f2\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.738860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.738829    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:26.779828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:26.779761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:27.028328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:27.028297    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"]
Nov 01 20:17:27.031867 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:17:27.031839    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2b092f_b70b_476e_97ab_bb937ad067e9.slice/crio-5c7f996368360dce55be505bc438dfdcd58d9066642c07cb3b9c6d41e1dccf0c WatchSource:0}: Error finding container 5c7f996368360dce55be505bc438dfdcd58d9066642c07cb3b9c6d41e1dccf0c: Status 404 returned error can't find the container with id 5c7f996368360dce55be505bc438dfdcd58d9066642c07cb3b9c6d41e1dccf0c
Nov 01 20:17:27.668850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:27.668808    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:27.768476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:27.768443    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:27.982249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:27.982215    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:28.075482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:28.074220    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" event={"ID":"5d2b092f-b70b-476e-97ab-bb937ad067e9","Type":"ContainerStarted","Data":"8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72"}
Nov 01 20:17:28.075482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:28.074288    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" event={"ID":"5d2b092f-b70b-476e-97ab-bb937ad067e9","Type":"ContainerStarted","Data":"5c7f996368360dce55be505bc438dfdcd58d9066642c07cb3b9c6d41e1dccf0c"}
Nov 01 20:17:28.075482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:28.075399    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:28.141796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:28.123043    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" podStartSLOduration=2.123020954 podStartE2EDuration="2.123020954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:28.094108396 +0000 UTC m=+1093.509740870" watchObservedRunningTime="2026-11-01 20:17:28.123020954 +0000 UTC m=+1093.538653428"
Nov 01 20:17:28.777161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:28.777126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:28.902738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:28.902669    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:29.135331 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.135239    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:29.135331 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.135272    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:29.135331 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.135330    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials podName:5d2b092f-b70b-476e-97ab-bb937ad067e9 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:29.635314392 +0000 UTC m=+1095.050946862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9") : secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:29.172403 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.172358    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:17:29.192171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.192136    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"]
Nov 01 20:17:29.224080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.224049    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"]
Nov 01 20:17:29.228848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.228820    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.236135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.236106    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"]
Nov 01 20:17:29.239518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.239492    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\""
Nov 01 20:17:29.279706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.279628    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:29.342291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.342245    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.342878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.342844    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.343082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.343062    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4hb\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.343334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.343319    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.445300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.445272    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.445472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.445359    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.445522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.445498    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.445522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.445516    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4hb\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.446155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.446125    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.446277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.446211    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.447742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.447721    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.453216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.453190    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4hb\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.547806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.547765    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:29.648874 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.648843    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:29.648874 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.648869    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:29.649064 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:29.648922    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials podName:5d2b092f-b70b-476e-97ab-bb937ad067e9 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:30.648905431 +0000 UTC m=+1096.064537886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9") : secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:29.841701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:29.841667    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"]
Nov 01 20:17:29.842955 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:17:29.842919    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode513bfca_0cd4_4914_9b11_bb153a143c2c.slice/crio-49202049725d8f8af13d12880768a2fda4f4e07cca162fb0ef9248b98f571f1a WatchSource:0}: Error finding container 49202049725d8f8af13d12880768a2fda4f4e07cca162fb0ef9248b98f571f1a: Status 404 returned error can't find the container with id 49202049725d8f8af13d12880768a2fda4f4e07cca162fb0ef9248b98f571f1a
Nov 01 20:17:30.037349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:30.037305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:30.110899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:30.108744    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk" event={"ID":"e513bfca-0cd4-4914-9b11-bb153a143c2c","Type":"ContainerStarted","Data":"c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5"}
Nov 01 20:17:30.110899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:30.108814    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk" event={"ID":"e513bfca-0cd4-4914-9b11-bb153a143c2c","Type":"ContainerStarted","Data":"49202049725d8f8af13d12880768a2fda4f4e07cca162fb0ef9248b98f571f1a"}
Nov 01 20:17:30.110899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:30.108890    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:30.159501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:30.157868    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk" podStartSLOduration=1.157842821 podStartE2EDuration="1.157842821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:30.147938345 +0000 UTC m=+1095.563570855" watchObservedRunningTime="2026-11-01 20:17:30.157842821 +0000 UTC m=+1095.573475362"
Nov 01 20:17:30.672338 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:30.672298    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:30.672338 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:30.672336    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:30.672633 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:30.672407    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials podName:5d2b092f-b70b-476e-97ab-bb937ad067e9 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:32.67238707 +0000 UTC m=+1098.088019520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9") : secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:30.728297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:30.728266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:31.339821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:31.339784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:32.080575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:32.080528    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:32.710420 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:32.710395    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:32.710420 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:32.710416    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp: secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:32.710860 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:32.710469    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials podName:5d2b092f-b70b-476e-97ab-bb937ad067e9 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:36.710454585 +0000 UTC m=+1102.126087037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9") : secret "agc-cred-b2b285556e64" not found
Nov 01 20:17:32.839510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:32.839476    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:33.466727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:33.466694    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:33.593608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:33.593572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:34.235074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:34.235039    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:34.953045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:34.953006    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:35.732013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:35.731966    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:36.114874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.114767    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" podUID="5d2b092f-b70b-476e-97ab-bb937ad067e9" containerName="payload-processing" containerID="cri-o://8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72" gracePeriod=25
Nov 01 20:17:36.410194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.410161    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:36.478207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.478173    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:36.496588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.496553    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2\") pod \"5d2b092f-b70b-476e-97ab-bb937ad067e9\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") "
Nov 01 20:17:36.496785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.496690    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials\") pod \"5d2b092f-b70b-476e-97ab-bb937ad067e9\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") "
Nov 01 20:17:36.496860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.496791    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume\") pod \"5d2b092f-b70b-476e-97ab-bb937ad067e9\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") "
Nov 01 20:17:36.496979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.496963    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume\") pod \"5d2b092f-b70b-476e-97ab-bb937ad067e9\" (UID: \"5d2b092f-b70b-476e-97ab-bb937ad067e9\") "
Nov 01 20:17:36.497238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.497208    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume" pod "5d2b092f-b70b-476e-97ab-bb937ad067e9" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:36.497327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.497295    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume" pod "5d2b092f-b70b-476e-97ab-bb937ad067e9" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:36.497577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.497554    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:36.497577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.497577    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d2b092f-b70b-476e-97ab-bb937ad067e9-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:36.499524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.499504    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2" pod "5d2b092f-b70b-476e-97ab-bb937ad067e9" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9"). InnerVolumeSpecName "kube-api-access-d52f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:36.499613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.499535    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials" pod "5d2b092f-b70b-476e-97ab-bb937ad067e9" (UID: "5d2b092f-b70b-476e-97ab-bb937ad067e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:36.606669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.604037    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52f2\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-kube-api-access-d52f2\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:36.606669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:36.604079    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d2b092f-b70b-476e-97ab-bb937ad067e9-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:37.167434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.167336    6479 generic.go:356] "Generic (PLEG): container finished" podID="5d2b092f-b70b-476e-97ab-bb937ad067e9" containerID="8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72" exitCode=0
Nov 01 20:17:37.167434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.167404    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"
Nov 01 20:17:37.167938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.167429    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" event={"ID":"5d2b092f-b70b-476e-97ab-bb937ad067e9","Type":"ContainerDied","Data":"8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72"}
Nov 01 20:17:37.167938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.167495    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp" event={"ID":"5d2b092f-b70b-476e-97ab-bb937ad067e9","Type":"ContainerDied","Data":"5c7f996368360dce55be505bc438dfdcd58d9066642c07cb3b9c6d41e1dccf0c"}
Nov 01 20:17:37.167938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.167527    6479 scope.go:122] "RemoveContainer" containerID="8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72"
Nov 01 20:17:37.179634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.179604    6479 scope.go:122] "RemoveContainer" containerID="8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72"
Nov 01 20:17:37.180031 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:37.179999    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72\": container with ID starting with 8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72 not found: ID does not exist" containerID="8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72"
Nov 01 20:17:37.180120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.180043    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72"} err="failed to get container status \"8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72\": rpc error: code = NotFound desc = could not find container \"8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72\": container with ID starting with 8096f9305f0d5c751c1a563c007c6b557e9d7e06c3b0a585080c211f15fdfa72 not found: ID does not exist"
Nov 01 20:17:37.187383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.187354    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"]
Nov 01 20:17:37.190377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.190350    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55q79cp"]
Nov 01 20:17:37.329130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:37.329097    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:38.179730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:38.179696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:38.709553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:38.709522    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2b092f-b70b-476e-97ab-bb937ad067e9" path="/var/lib/kubelet/pods/5d2b092f-b70b-476e-97ab-bb937ad067e9/volumes"
Nov 01 20:17:38.947311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:38.947274    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:39.674534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:39.674496    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:40.449866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:40.449829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:41.117782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:41.117752    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:17:41.199533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:41.199476    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:41.830116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:41.830085    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:17:41.881852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:41.881817    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:42.796422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:42.796361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:43.725974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:43.725939    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:44.573676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:44.573631    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:45.561303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:45.561264    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:46.387947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:46.387907    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:47.243092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:47.243053    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:48.058315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.058275    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:48.682245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.682189    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:48.836012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.835980    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"]
Nov 01 20:17:48.837259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.837232    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5d2b092f-b70b-476e-97ab-bb937ad067e9" containerName="payload-processing"
Nov 01 20:17:48.837408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.837395    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2b092f-b70b-476e-97ab-bb937ad067e9" containerName="payload-processing"
Nov 01 20:17:48.837707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.837691    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2b092f-b70b-476e-97ab-bb937ad067e9" containerName="payload-processing"
Nov 01 20:17:48.843598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.843576    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.857602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.857572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:48.858106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.858078    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.858263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.858250    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.858330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.858293    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g68\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.858380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.858350    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.860206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.859930    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"]
Nov 01 20:17:48.984674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.981492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:48.999669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.985289    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.999669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.986250    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g68\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.999669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.986788    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.999669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.986897    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.999669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.987112    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:48.999669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:48.997266    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:49.028947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:49.028912    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:49.031549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:49.031519    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g68\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:49.191740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:49.191691    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:49.441749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:49.437430    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"]
Nov 01 20:17:49.441749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:49.437641    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g" podUID="211ddc12-81e0-4492-a7ac-32458b2f6f62" containerName="sim" containerID="cri-o://cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910" gracePeriod=30
Nov 01 20:17:49.590824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:49.589211    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:50.055967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.055925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:50.082896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.082864    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:17:50.166308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.166262    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"]
Nov 01 20:17:50.185667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.185434    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh\" (UniqueName: \"kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh\") pod \"211ddc12-81e0-4492-a7ac-32458b2f6f62\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") "
Nov 01 20:17:50.186708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.185856    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls\" (UniqueName: \"kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls\") pod \"211ddc12-81e0-4492-a7ac-32458b2f6f62\" (UID: \"211ddc12-81e0-4492-a7ac-32458b2f6f62\") "
Nov 01 20:17:50.223723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.223585    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls" pod "211ddc12-81e0-4492-a7ac-32458b2f6f62" (UID: "211ddc12-81e0-4492-a7ac-32458b2f6f62"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:17:50.243202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.243145    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh" pod "211ddc12-81e0-4492-a7ac-32458b2f6f62" (UID: "211ddc12-81e0-4492-a7ac-32458b2f6f62"). InnerVolumeSpecName "kube-api-access-8n7fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:50.288601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.288572    6479 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/211ddc12-81e0-4492-a7ac-32458b2f6f62-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:50.288601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.288600    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7fh\" (UniqueName: \"kubernetes.io/projected/211ddc12-81e0-4492-a7ac-32458b2f6f62-kube-api-access-8n7fh\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:50.342962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.342920    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" event={"ID":"d1d85f5f-d19f-4521-8e6f-d4da2470cce1","Type":"ContainerStarted","Data":"92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596"}
Nov 01 20:17:50.343158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.342991    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" event={"ID":"d1d85f5f-d19f-4521-8e6f-d4da2470cce1","Type":"ContainerStarted","Data":"7df78be9510b656606209343c9f447796ec766c0af232d3e1047b86de8fa65a7"}
Nov 01 20:17:50.343158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.343109    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:50.344325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.344300    6479 generic.go:356] "Generic (PLEG): container finished" podID="211ddc12-81e0-4492-a7ac-32458b2f6f62" containerID="cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910" exitCode=0
Nov 01 20:17:50.344446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.344356    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"
Nov 01 20:17:50.344446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.344383    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g" event={"ID":"211ddc12-81e0-4492-a7ac-32458b2f6f62","Type":"ContainerDied","Data":"cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910"}
Nov 01 20:17:50.344446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.344429    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g" event={"ID":"211ddc12-81e0-4492-a7ac-32458b2f6f62","Type":"ContainerDied","Data":"76c1dd3298ae479c25bc3c4867086c144cfb2009cdda73eb40a777f46294cec9"}
Nov 01 20:17:50.344605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.344452    6479 scope.go:122] "RemoveContainer" containerID="cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910"
Nov 01 20:17:50.363639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.363581    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" podStartSLOduration=2.36356255 podStartE2EDuration="2.36356255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:50.358100585 +0000 UTC m=+1115.773733059" watchObservedRunningTime="2026-11-01 20:17:50.36356255 +0000 UTC m=+1115.779195024"
Nov 01 20:17:50.371787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.371750    6479 scope.go:122] "RemoveContainer" containerID="cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910"
Nov 01 20:17:50.372442 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:50.372252    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910\": container with ID starting with cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910 not found: ID does not exist" containerID="cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910"
Nov 01 20:17:50.372442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.372347    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910"} err="failed to get container status \"cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910\": rpc error: code = NotFound desc = could not find container \"cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910\": container with ID starting with cbbab589d17231152d34d2e2602250620ae92ca3c7480658ada8bd27960d3910 not found: ID does not exist"
Nov 01 20:17:50.384255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.384226    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"]
Nov 01 20:17:50.388027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.388006    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-8d9fe7/e2e-usage-sim-71a622-65f44b5db4-9597g"]
Nov 01 20:17:50.708671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.708624    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211ddc12-81e0-4492-a7ac-32458b2f6f62" path="/var/lib/kubelet/pods/211ddc12-81e0-4492-a7ac-32458b2f6f62/volumes"
Nov 01 20:17:50.907385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:50.907344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:51.008417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.008325    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:51.065411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.065379    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"]
Nov 01 20:17:51.113668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.113593    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:51.113860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.113736    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"]
Nov 01 20:17:51.115675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.114875    6479 cpu_manager.go:414] "containerMap: removing container" podUID="211ddc12-81e0-4492-a7ac-32458b2f6f62" containerName="sim"
Nov 01 20:17:51.115675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.114901    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211ddc12-81e0-4492-a7ac-32458b2f6f62" containerName="sim"
Nov 01 20:17:51.115675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.115121    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="211ddc12-81e0-4492-a7ac-32458b2f6f62" containerName="sim"
Nov 01 20:17:51.127487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.126667    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.149038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.147995    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"]
Nov 01 20:17:51.294785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.206545    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.294785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.206623    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65pz\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.294785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.206868    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.294785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.207009    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.316730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.316335    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.317564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.317541    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.317912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.317891    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.318059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.318043    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65pz\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.323995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.317385    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.329499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.325134    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.364796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.353264    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65pz\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.382905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.382839    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.420087 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:51.420054    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-29b57c/agc-cred-a9325faf1444: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:51.420087 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:51.420082    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:51.420331 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:51.420140    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials podName:d1d85f5f-d19f-4521-8e6f-d4da2470cce1 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:51.920120166 +0000 UTC m=+1117.335752637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1") : secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:51.604729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:51.594999    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:51.973065 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:51.973029    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-29b57c/agc-cred-a9325faf1444: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:51.973718 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:51.973264    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:51.973718 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:51.973336    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials podName:d1d85f5f-d19f-4521-8e6f-d4da2470cce1 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:52.973316596 +0000 UTC m=+1118.388949064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1") : secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:52.209442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.209406    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"]
Nov 01 20:17:52.210500 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:17:52.210470    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a812390_893b_405b_add5_b3f8dd23e4bb.slice/crio-2c47a2cd8dea702758b93f787d5a1604a4ee160005a2392bb0286b91f97fd70b WatchSource:0}: Error finding container 2c47a2cd8dea702758b93f787d5a1604a4ee160005a2392bb0286b91f97fd70b: Status 404 returned error can't find the container with id 2c47a2cd8dea702758b93f787d5a1604a4ee160005a2392bb0286b91f97fd70b
Nov 01 20:17:52.245252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.245214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:52.424712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.422350    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc" event={"ID":"5a812390-893b-405b-add5-b3f8dd23e4bb","Type":"ContainerStarted","Data":"2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d"}
Nov 01 20:17:52.424712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.422421    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc" event={"ID":"5a812390-893b-405b-add5-b3f8dd23e4bb","Type":"ContainerStarted","Data":"2c47a2cd8dea702758b93f787d5a1604a4ee160005a2392bb0286b91f97fd70b"}
Nov 01 20:17:52.424712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.422452    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:17:52.444013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.443963    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc" podStartSLOduration=1.443945609 podStartE2EDuration="1.443945609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:52.439321608 +0000 UTC m=+1117.854954105" watchObservedRunningTime="2026-11-01 20:17:52.443945609 +0000 UTC m=+1117.859578082"
Nov 01 20:17:52.952955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:52.952916    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:52.985716 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:52.985681    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-29b57c/agc-cred-a9325faf1444: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:52.985716 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:52.985711    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:52.985968 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:52.985778    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials podName:d1d85f5f-d19f-4521-8e6f-d4da2470cce1 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:54.985759882 +0000 UTC m=+1120.401392347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1") : secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:53.545364 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:53.545332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:54.202249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:54.202206    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:54.880354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:54.880310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:55.020695 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:55.019991    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-29b57c/agc-cred-a9325faf1444: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:55.020695 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:55.020124    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v: secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:55.020695 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:55.020192    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials podName:d1d85f5f-d19f-4521-8e6f-d4da2470cce1 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:59.020169798 +0000 UTC m=+1124.435802253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1") : secret "agc-cred-a9325faf1444" not found
Nov 01 20:17:55.581277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:55.581241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:56.279035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:56.278998    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:56.954359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:56.954325    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:57.422047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.421950    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" podUID="d1d85f5f-d19f-4521-8e6f-d4da2470cce1" containerName="payload-processing" containerID="cri-o://92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596" gracePeriod=25
Nov 01 20:17:57.581062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.581021    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:57.798744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.798715    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:57.884794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.884756    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68\") pod \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") "
Nov 01 20:17:57.884978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.884821    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume\") pod \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") "
Nov 01 20:17:57.884978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.884854    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume\") pod \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") "
Nov 01 20:17:57.884978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.884917    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials\") pod \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\" (UID: \"d1d85f5f-d19f-4521-8e6f-d4da2470cce1\") "
Nov 01 20:17:57.885302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.885270    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume" pod "d1d85f5f-d19f-4521-8e6f-d4da2470cce1" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:57.885302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.885285    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume" pod "d1d85f5f-d19f-4521-8e6f-d4da2470cce1" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:57.885576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.885555    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:57.885668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.885580    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:57.887113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.887091    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68" pod "d1d85f5f-d19f-4521-8e6f-d4da2470cce1" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1"). InnerVolumeSpecName "kube-api-access-65g68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:57.887215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.887193    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials" pod "d1d85f5f-d19f-4521-8e6f-d4da2470cce1" (UID: "d1d85f5f-d19f-4521-8e6f-d4da2470cce1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:57.987017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.986971    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g68\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-kube-api-access-65g68\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:57.987017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:57.987010    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d85f5f-d19f-4521-8e6f-d4da2470cce1-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:58.289461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.289424    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:58.481533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.481493    6479 generic.go:356] "Generic (PLEG): container finished" podID="d1d85f5f-d19f-4521-8e6f-d4da2470cce1" containerID="92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596" exitCode=0
Nov 01 20:17:58.482000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.481538    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"
Nov 01 20:17:58.482000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.481579    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" event={"ID":"d1d85f5f-d19f-4521-8e6f-d4da2470cce1","Type":"ContainerDied","Data":"92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596"}
Nov 01 20:17:58.482000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.481714    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v" event={"ID":"d1d85f5f-d19f-4521-8e6f-d4da2470cce1","Type":"ContainerDied","Data":"7df78be9510b656606209343c9f447796ec766c0af232d3e1047b86de8fa65a7"}
Nov 01 20:17:58.482000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.481748    6479 scope.go:122] "RemoveContainer" containerID="92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596"
Nov 01 20:17:58.494679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.494655    6479 scope.go:122] "RemoveContainer" containerID="92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596"
Nov 01 20:17:58.495008 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:58.494986    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596\": container with ID starting with 92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596 not found: ID does not exist" containerID="92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596"
Nov 01 20:17:58.495085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.495020    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596"} err="failed to get container status \"92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596\": rpc error: code = NotFound desc = could not find container \"92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596\": container with ID starting with 92b6d76219b2eb9c2f27378a8925dc2fb23ca206cb0c4c24fbf8b3eeefef7596 not found: ID does not exist"
Nov 01 20:17:58.504908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.504880    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"]
Nov 01 20:17:58.507946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.507924    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-5458lk98v"]
Nov 01 20:17:58.708995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.708963    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d85f5f-d19f-4521-8e6f-d4da2470cce1" path="/var/lib/kubelet/pods/d1d85f5f-d19f-4521-8e6f-d4da2470cce1/volumes"
Nov 01 20:17:58.984722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:58.984606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:17:59.182433 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:17:59.182390    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:17:59.785088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:17:59.785050    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:00.552302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:00.552249    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:01.159828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:01.159789    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:01.883901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:01.883867    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:02.617400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:02.617346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:03.200418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:03.200383    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:03.431158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:03.431131    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:18:03.482167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:03.482058    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"]
Nov 01 20:18:03.865298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:03.865186    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:03.926472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:03.926438    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"]
Nov 01 20:18:04.668923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:04.668885    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:05.418960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:05.418924    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:06.048021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.047989    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:06.643736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.643700    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"]
Nov 01 20:18:06.644501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.644479    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d1d85f5f-d19f-4521-8e6f-d4da2470cce1" containerName="payload-processing"
Nov 01 20:18:06.644501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.644500    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d85f5f-d19f-4521-8e6f-d4da2470cce1" containerName="payload-processing"
Nov 01 20:18:06.644639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.644629    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d85f5f-d19f-4521-8e6f-d4da2470cce1" containerName="payload-processing"
Nov 01 20:18:06.648786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.648766    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.658435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.658407    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-9be196-openshift-default-dockercfg-wrldh\""
Nov 01 20:18:06.661854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.661830    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"]
Nov 01 20:18:06.662856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.662793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:06.794282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.794243    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.794505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.794487    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.794772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.794733    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.794973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.794956    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.795092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.795079    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.795199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.795186    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.795367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.795352    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.795496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.795481    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljhz\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.795637 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.795620    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.896907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.896810    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.896907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.896861    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.896938    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.896993    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897025    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897059    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897231    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897262    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljhz\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897322    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897859    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.897909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.897890    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.898098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.898031    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.898335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.898306    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.898456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.898386    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.899522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.899505    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.900019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.899998    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.904582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.904554    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.904743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.904725    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljhz\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:06.967354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:06.967317    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:07.241245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.241214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:07.253607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.253575    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"]
Nov 01 20:18:07.254264 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:18:07.254237    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b718eed_0b7f_48f0_9a27_a03bbd79850c.slice/crio-2b48c7328e96368decdf098c2da1325682b9f28929fb957e0ff0706d4815e427 WatchSource:0}: Error finding container 2b48c7328e96368decdf098c2da1325682b9f28929fb957e0ff0706d4815e427: Status 404 returned error can't find the container with id 2b48c7328e96368decdf098c2da1325682b9f28929fb957e0ff0706d4815e427
Nov 01 20:18:07.257554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.257521    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:18:07.257669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.257588    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:18:07.257669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.257615    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:18:07.587786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.587689    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" event={"ID":"6b718eed-0b7f-48f0-9a27-a03bbd79850c","Type":"ContainerStarted","Data":"2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020"}
Nov 01 20:18:07.587786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.587759    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" event={"ID":"6b718eed-0b7f-48f0-9a27-a03bbd79850c","Type":"ContainerStarted","Data":"2b48c7328e96368decdf098c2da1325682b9f28929fb957e0ff0706d4815e427"}
Nov 01 20:18:07.606579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.606531    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podStartSLOduration=1.606518305 podStartE2EDuration="1.606518305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:07.605149811 +0000 UTC m=+1133.020782285" watchObservedRunningTime="2026-11-01 20:18:07.606518305 +0000 UTC m=+1133.022150778"
Nov 01 20:18:07.975679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.972757    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:07.977285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:07.977250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:08.482688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:08.482621    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7" podUID="72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" containerName="payload-processing" containerID="cri-o://77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0" gracePeriod=25
Nov 01 20:18:08.637063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:08.637031    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:08.928750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:08.927160    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq" podUID="53466176-a2d9-4641-9afb-95f586f051f5" containerName="payload-processing" containerID="cri-o://1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc" gracePeriod=25
Nov 01 20:18:08.989987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:08.989915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:08.991248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:08.990772    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 20:18:08.991248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:08.990812    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 20:18:09.086503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.086464    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:09.093751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.093726    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:18:09.189726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.189070    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv\") pod \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") "
Nov 01 20:18:09.189726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.189156    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume\") pod \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") "
Nov 01 20:18:09.189726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.189360    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials\") pod \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") "
Nov 01 20:18:09.189726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.189412    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume\") pod \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\" (UID: \"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9\") "
Nov 01 20:18:09.190442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.190130    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume" pod "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" (UID: "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:09.193671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.190919    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.193671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.191623    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume" pod "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" (UID: "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:09.205758 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.205730    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv" pod "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" (UID: "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9"). InnerVolumeSpecName "kube-api-access-9qvsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:09.206332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.206304    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials" pod "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" (UID: "72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:09.300681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.297879    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.300681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.297920    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.300681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.297934    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvsv\" (UniqueName: \"kubernetes.io/projected/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9-kube-api-access-9qvsv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.537355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.537305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:09.551046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.550713    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:18:09.619866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.618883    6479 generic.go:356] "Generic (PLEG): container finished" podID="53466176-a2d9-4641-9afb-95f586f051f5" containerID="1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc" exitCode=0
Nov 01 20:18:09.619866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.619163    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq" event={"ID":"53466176-a2d9-4641-9afb-95f586f051f5","Type":"ContainerDied","Data":"1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc"}
Nov 01 20:18:09.619866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.619232    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq" event={"ID":"53466176-a2d9-4641-9afb-95f586f051f5","Type":"ContainerDied","Data":"3e316f5e9f041919bcc022fa19907611235f9c515b8db1b42bc40660db0f4a58"}
Nov 01 20:18:09.619866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.619310    6479 scope.go:122] "RemoveContainer" containerID="1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc"
Nov 01 20:18:09.619866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.619497    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"
Nov 01 20:18:09.623280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.623249    6479 generic.go:356] "Generic (PLEG): container finished" podID="72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" containerID="77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0" exitCode=0
Nov 01 20:18:09.623686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.623583    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"
Nov 01 20:18:09.624437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.624411    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7" event={"ID":"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9","Type":"ContainerDied","Data":"77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0"}
Nov 01 20:18:09.625533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.625357    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7" event={"ID":"72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9","Type":"ContainerDied","Data":"74874367b6baf02c87c163b40bd0e4ba573b35c9cc85403e365f55d3a8bd420a"}
Nov 01 20:18:09.638665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.633225    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:09.638665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.636328    6479 scope.go:122] "RemoveContainer" containerID="1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc"
Nov 01 20:18:09.639339 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:09.639308    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc\": container with ID starting with 1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc not found: ID does not exist" containerID="1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc"
Nov 01 20:18:09.639437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.639351    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc"} err="failed to get container status \"1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc\": rpc error: code = NotFound desc = could not find container \"1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc\": container with ID starting with 1c34d619d761f12b618831f2fd97a1a354171c3fd227efbce85a74b3ea9d46bc not found: ID does not exist"
Nov 01 20:18:09.639437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.639374    6479 scope.go:122] "RemoveContainer" containerID="77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0"
Nov 01 20:18:09.668670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.654860    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f\") pod \"53466176-a2d9-4641-9afb-95f586f051f5\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") "
Nov 01 20:18:09.668670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.654962    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials\") pod \"53466176-a2d9-4641-9afb-95f586f051f5\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") "
Nov 01 20:18:09.668670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.655025    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume\") pod \"53466176-a2d9-4641-9afb-95f586f051f5\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") "
Nov 01 20:18:09.668670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.655324    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume\") pod \"53466176-a2d9-4641-9afb-95f586f051f5\" (UID: \"53466176-a2d9-4641-9afb-95f586f051f5\") "
Nov 01 20:18:09.687423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.686150    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials" pod "53466176-a2d9-4641-9afb-95f586f051f5" (UID: "53466176-a2d9-4641-9afb-95f586f051f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:09.694765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.693638    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume" pod "53466176-a2d9-4641-9afb-95f586f051f5" (UID: "53466176-a2d9-4641-9afb-95f586f051f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:09.695187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.695125    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume" pod "53466176-a2d9-4641-9afb-95f586f051f5" (UID: "53466176-a2d9-4641-9afb-95f586f051f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:09.703980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.699784    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"]
Nov 01 20:18:09.720784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.720760    6479 scope.go:122] "RemoveContainer" containerID="77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0"
Nov 01 20:18:09.721431 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:09.721409    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0\": container with ID starting with 77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0 not found: ID does not exist" containerID="77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0"
Nov 01 20:18:09.721598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.721573    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0"} err="failed to get container status \"77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0\": rpc error: code = NotFound desc = could not find container \"77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0\": container with ID starting with 77f5d6f9c8c269b9836e760adad7e16be4b1ac5bf6db981c225195fd538f90d0 not found: ID does not exist"
Nov 01 20:18:09.766111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.766067    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.766351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.766333    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.766459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.766445    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53466176-a2d9-4641-9afb-95f586f051f5-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.819789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.817466    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-844786ll7"]
Nov 01 20:18:09.891056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.881870    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f" pod "53466176-a2d9-4641-9afb-95f586f051f5" (UID: "53466176-a2d9-4641-9afb-95f586f051f5"). InnerVolumeSpecName "kube-api-access-c6h6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:09.973915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.973872    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6h6f\" (UniqueName: \"kubernetes.io/projected/53466176-a2d9-4641-9afb-95f586f051f5-kube-api-access-c6h6f\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:09.974595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.974567    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 20:18:09.974876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:09.974847    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 20:18:10.036095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.035812    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"]
Nov 01 20:18:10.041064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.041024    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-55482wsnq"]
Nov 01 20:18:10.374672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.370048    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:10.708307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.708276    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53466176-a2d9-4641-9afb-95f586f051f5" path="/var/lib/kubelet/pods/53466176-a2d9-4641-9afb-95f586f051f5/volumes"
Nov 01 20:18:10.708759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.708744    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" path="/var/lib/kubelet/pods/72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9/volumes"
Nov 01 20:18:10.968099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.968008    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 20:18:10.968377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:10.968347    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 20:18:11.145870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:11.145836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:11.748125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:11.748094    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:11.970039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:11.970004    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 20:18:11.970285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:11.970261    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 20:18:12.519169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:12.519135    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:12.971034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:12.970997    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 20:18:12.971346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:12.971316    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 20:18:13.150982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.150948    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:13.510294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.510244    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:13.606666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.606444    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.618698    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"]
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.619895    6479 cpu_manager.go:414] "containerMap: removing container" podUID="72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" containerName="payload-processing"
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.619918    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" containerName="payload-processing"
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.619947    6479 cpu_manager.go:414] "containerMap: removing container" podUID="53466176-a2d9-4641-9afb-95f586f051f5" containerName="payload-processing"
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.619957    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53466176-a2d9-4641-9afb-95f586f051f5" containerName="payload-processing"
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.620148    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="53466176-a2d9-4641-9afb-95f586f051f5" containerName="payload-processing"
Nov 01 20:18:13.630801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.620177    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ff5d09-a7c3-45ff-b07a-a81af4cc6ae9" containerName="payload-processing"
Nov 01 20:18:13.703671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.699205    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:13.707669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.706035    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"]
Nov 01 20:18:13.707669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.706955    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-646b8d4cafa4\""
Nov 01 20:18:13.737671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.726196    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bccce1\""
Nov 01 20:18:13.737671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.726573    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"routing-overlay\""
Nov 01 20:18:13.737671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.726577    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"openshift-service-ca.crt\""
Nov 01 20:18:13.806687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.806578    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-e2e--900891f9-dockercfg-ns4ld\""
Nov 01 20:18:13.845674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.829563    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"kube-root-ca.crt\""
Nov 01 20:18:13.915295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.915259    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:13.915474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.915445    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csg6j\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:13.915520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.915497    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:13.915588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.915563    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:13.968541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.968504    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 20:18:13.968754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:13.968556    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 20:18:14.016932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.016889    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg6j\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.017134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.016987    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.017134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.017015    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.017249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.017130    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.017883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.017857    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.018027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.018002    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.019776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.019748    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.024735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.024714    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg6j\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.176720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.169041    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:14.357923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.357280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:14.820238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.820212    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"]
Nov 01 20:18:14.820997 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:18:14.820971    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb782ebe9_894a_4ca9_bab1_c3bace367c5e.slice/crio-1ad96b33bbb645ab921d0773b6f01e78a8bfdc52f4edf580ee13520de4b1ccc6 WatchSource:0}: Error finding container 1ad96b33bbb645ab921d0773b6f01e78a8bfdc52f4edf580ee13520de4b1ccc6: Status 404 returned error can't find the container with id 1ad96b33bbb645ab921d0773b6f01e78a8bfdc52f4edf580ee13520de4b1ccc6
Nov 01 20:18:14.875688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.875611    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld" event={"ID":"b782ebe9-894a-4ca9-bab1-c3bace367c5e","Type":"ContainerStarted","Data":"1ad96b33bbb645ab921d0773b6f01e78a8bfdc52f4edf580ee13520de4b1ccc6"}
Nov 01 20:18:14.935159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.935123    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:14.974238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.974211    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:14.974717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.974691    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:14.975483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:14.975463    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:18:15.044143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:15.028084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:15.654511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:15.654477    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:15.884631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:15.884594    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld" event={"ID":"b782ebe9-894a-4ca9-bab1-c3bace367c5e","Type":"ContainerStarted","Data":"2336a810e73aacaceac0a958cccab75980c6f183a882e3f478c98cae0a98ba76"}
Nov 01 20:18:15.885087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:15.884698    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:15.903402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:15.903350    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld" podStartSLOduration=2.903336628 podStartE2EDuration="2.903336628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:15.900884508 +0000 UTC m=+1141.316516996" watchObservedRunningTime="2026-11-01 20:18:15.903336628 +0000 UTC m=+1141.318969324"
Nov 01 20:18:16.371006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:16.370974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:17.026613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:17.026579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:17.643635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:17.643610    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:18.192986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:18.192939    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:18.658130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:18.658079    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:18:18.851090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:18.851059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:19.536252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:19.536220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:20.404273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:20.404231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:21.207670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:21.205283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:22.229224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:22.229172    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:22.422683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:22.422320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:23.252625 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:23.252464    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:23.841234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:23.841202    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:23.949382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:23.949348    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:24.152603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.152471    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:24.436821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.436744    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"]
Nov 01 20:18:24.442739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.442717    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.446407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.446386    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-8876b9054304\""
Nov 01 20:18:24.451600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.451571    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"]
Nov 01 20:18:24.555895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.555842    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xdf\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.556096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.555902    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.556096 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.556048    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.556245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.556227    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.657837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.657791    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.658051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.657942    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xdf\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.658051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.657976    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.658051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.658034    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.658833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.658806    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.658963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.658847    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.660658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.660606    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.665854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.665827    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xdf\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.760400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.760358    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:24.843559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:24.843199    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:25.050508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:25.050481    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"]
Nov 01 20:18:25.051057 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:18:25.051029    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e04449e_185a_43be_9f41_72505bb8a121.slice/crio-fd512f78b5ab60773a77eeb6f2c9aa6b411ae37e170a52ad2b5d09bfa9903363 WatchSource:0}: Error finding container fd512f78b5ab60773a77eeb6f2c9aa6b411ae37e170a52ad2b5d09bfa9903363: Status 404 returned error can't find the container with id fd512f78b5ab60773a77eeb6f2c9aa6b411ae37e170a52ad2b5d09bfa9903363
Nov 01 20:18:25.119573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:25.119541    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz" event={"ID":"3e04449e-185a-43be-9f41-72505bb8a121","Type":"ContainerStarted","Data":"fd512f78b5ab60773a77eeb6f2c9aa6b411ae37e170a52ad2b5d09bfa9903363"}
Nov 01 20:18:25.529591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:25.529555    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:26.127674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:26.127625    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz" event={"ID":"3e04449e-185a-43be-9f41-72505bb8a121","Type":"ContainerStarted","Data":"1c3330d4ce9a91d629e20d62668594de368a40034f1018c272f762dc9c3ef4dd"}
Nov 01 20:18:26.127855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:26.127700    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:26.145068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:26.145016    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz" podStartSLOduration=2.145001496 podStartE2EDuration="2.145001496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:26.142845108 +0000 UTC m=+1151.558477606" watchObservedRunningTime="2026-11-01 20:18:26.145001496 +0000 UTC m=+1151.560634000"
Nov 01 20:18:26.237892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:26.237859    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:26.269115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:26.269083    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"]
Nov 01 20:18:26.903593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:26.903565    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:27.200900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:27.200871    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:28.070840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:28.070807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:28.651373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:28.651329    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:29.175344 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:29.175303    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:18:29.260432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:29.260396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:29.838881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:29.838846    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:30.465741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:30.465701    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:31.146340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.146298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:31.278099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.277380    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc" podUID="5a812390-893b-405b-add5-b3f8dd23e4bb" containerName="payload-processing" containerID="cri-o://2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d" gracePeriod=25
Nov 01 20:18:31.539028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.538964    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:31.573255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.573232    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:18:31.591455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.591407    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials\") pod \"5a812390-893b-405b-add5-b3f8dd23e4bb\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") "
Nov 01 20:18:31.591927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.591896    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz\") pod \"5a812390-893b-405b-add5-b3f8dd23e4bb\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") "
Nov 01 20:18:31.592053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.592009    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume\") pod \"5a812390-893b-405b-add5-b3f8dd23e4bb\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") "
Nov 01 20:18:31.592053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.592046    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume\") pod \"5a812390-893b-405b-add5-b3f8dd23e4bb\" (UID: \"5a812390-893b-405b-add5-b3f8dd23e4bb\") "
Nov 01 20:18:31.593060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.593000    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume" pod "5a812390-893b-405b-add5-b3f8dd23e4bb" (UID: "5a812390-893b-405b-add5-b3f8dd23e4bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:31.593478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.593437    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume" pod "5a812390-893b-405b-add5-b3f8dd23e4bb" (UID: "5a812390-893b-405b-add5-b3f8dd23e4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:31.594380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.594354    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials" pod "5a812390-893b-405b-add5-b3f8dd23e4bb" (UID: "5a812390-893b-405b-add5-b3f8dd23e4bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:31.597413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.597386    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz" pod "5a812390-893b-405b-add5-b3f8dd23e4bb" (UID: "5a812390-893b-405b-add5-b3f8dd23e4bb"). InnerVolumeSpecName "kube-api-access-w65pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:31.642747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.642717    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:31.694999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.694967    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:31.695127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.695000    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w65pz\" (UniqueName: \"kubernetes.io/projected/5a812390-893b-405b-add5-b3f8dd23e4bb-kube-api-access-w65pz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:31.695127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.695020    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:31.695127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.695034    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a812390-893b-405b-add5-b3f8dd23e4bb-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:31.940571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:31.940490    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:32.193884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.193793    6479 generic.go:356] "Generic (PLEG): container finished" podID="5a812390-893b-405b-add5-b3f8dd23e4bb" containerID="2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d" exitCode=0
Nov 01 20:18:32.193884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.193842    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"
Nov 01 20:18:32.194072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.193847    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc" event={"ID":"5a812390-893b-405b-add5-b3f8dd23e4bb","Type":"ContainerDied","Data":"2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d"}
Nov 01 20:18:32.194072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.193946    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc" event={"ID":"5a812390-893b-405b-add5-b3f8dd23e4bb","Type":"ContainerDied","Data":"2c47a2cd8dea702758b93f787d5a1604a4ee160005a2392bb0286b91f97fd70b"}
Nov 01 20:18:32.194072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.193966    6479 scope.go:122] "RemoveContainer" containerID="2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d"
Nov 01 20:18:32.217075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.217048    6479 scope.go:122] "RemoveContainer" containerID="2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d"
Nov 01 20:18:32.217377 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:32.217355    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d\": container with ID starting with 2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d not found: ID does not exist" containerID="2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d"
Nov 01 20:18:32.217457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.217385    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d"} err="failed to get container status \"2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d\": rpc error: code = NotFound desc = could not find container \"2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d\": container with ID starting with 2c681117902dc121bbc385e6fd6099811f635261a724c8beb3bb67df51dbd13d not found: ID does not exist"
Nov 01 20:18:32.234578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.234547    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"]
Nov 01 20:18:32.237519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.237498    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-7797vzvsc"]
Nov 01 20:18:32.710681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.710623    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a812390-893b-405b-add5-b3f8dd23e4bb" path="/var/lib/kubelet/pods/5a812390-893b-405b-add5-b3f8dd23e4bb/volumes"
Nov 01 20:18:32.742616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:32.742550    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:33.355727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:33.355692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:34.051881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:34.051841    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:34.649207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:34.649175    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:35.354789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:35.354748    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:35.948851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:35.948804    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:36.050510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:36.050478    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:36.166105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:36.166067    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:36.473717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:36.469102    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:36.495905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:36.476099    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"]
Nov 01 20:18:36.685665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:36.678180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:37.022243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:37.022208    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:37.150687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:37.149812    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:18:37.195746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:37.195705    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"]
Nov 01 20:18:37.654315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:37.654280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:38.256387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:38.256331    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:38.951351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:38.951316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:39.787258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:39.787219    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:40.815741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:40.815707    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:41.485671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:41.481709    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld" podUID="b782ebe9-894a-4ca9-bab1-c3bace367c5e" containerName="payload-processing" containerID="cri-o://2336a810e73aacaceac0a958cccab75980c6f183a882e3f478c98cae0a98ba76" gracePeriod=25
Nov 01 20:18:42.129695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.112713    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"]
Nov 01 20:18:42.129695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.113037    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy" containerID="cri-o://29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5" gracePeriod=30
Nov 01 20:18:42.152671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.151237    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"]
Nov 01 20:18:42.152671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.151556    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy" containerID="cri-o://3e93cab6a575d9f260e7e526a0bf2906a437cb74e526fb5910e25a025c233d15" gracePeriod=30
Nov 01 20:18:42.252974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.214640    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk" podUID="e513bfca-0cd4-4914-9b11-bb153a143c2c" containerName="payload-processing" containerID="cri-o://c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5" gracePeriod=25
Nov 01 20:18:42.260339 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.253619    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:42.760669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.752312    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:42.761222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.760720    6479 generic.go:356] "Generic (PLEG): container finished" podID="b782ebe9-894a-4ca9-bab1-c3bace367c5e" containerID="2336a810e73aacaceac0a958cccab75980c6f183a882e3f478c98cae0a98ba76" exitCode=0
Nov 01 20:18:42.802621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.802219    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld" event={"ID":"b782ebe9-894a-4ca9-bab1-c3bace367c5e","Type":"ContainerDied","Data":"2336a810e73aacaceac0a958cccab75980c6f183a882e3f478c98cae0a98ba76"}
Nov 01 20:18:42.802621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.802294    6479 scope.go:122] "RemoveContainer" containerID="2336a810e73aacaceac0a958cccab75980c6f183a882e3f478c98cae0a98ba76"
Nov 01 20:18:42.822149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.820587    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume\") pod \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") "
Nov 01 20:18:42.822149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.820796    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume\") pod \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") "
Nov 01 20:18:42.822149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.820850    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials\") pod \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") "
Nov 01 20:18:42.822149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.820951    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j\") pod \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\" (UID: \"b782ebe9-894a-4ca9-bab1-c3bace367c5e\") "
Nov 01 20:18:42.823306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.823119    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume" pod "b782ebe9-894a-4ca9-bab1-c3bace367c5e" (UID: "b782ebe9-894a-4ca9-bab1-c3bace367c5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:42.823306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.823279    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume" pod "b782ebe9-894a-4ca9-bab1-c3bace367c5e" (UID: "b782ebe9-894a-4ca9-bab1-c3bace367c5e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:42.848888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.848823    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials" pod "b782ebe9-894a-4ca9-bab1-c3bace367c5e" (UID: "b782ebe9-894a-4ca9-bab1-c3bace367c5e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:42.881968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.871261    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j" pod "b782ebe9-894a-4ca9-bab1-c3bace367c5e" (UID: "b782ebe9-894a-4ca9-bab1-c3bace367c5e"). InnerVolumeSpecName "kube-api-access-csg6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:42.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.924174    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csg6j\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-kube-api-access-csg6j\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:42.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.924217    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:42.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.924234    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b782ebe9-894a-4ca9-bab1-c3bace367c5e-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:42.924673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.924247    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b782ebe9-894a-4ca9-bab1-c3bace367c5e-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:42.974865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:42.974340    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:43.087975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.077860    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:43.167558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.167456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:43.189708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.188229    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:18:43.371040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.370956    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume\") pod \"e513bfca-0cd4-4914-9b11-bb153a143c2c\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") "
Nov 01 20:18:43.371222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371053    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume\") pod \"e513bfca-0cd4-4914-9b11-bb153a143c2c\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") "
Nov 01 20:18:43.371222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371143    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials\") pod \"e513bfca-0cd4-4914-9b11-bb153a143c2c\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") "
Nov 01 20:18:43.371348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371244    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb\") pod \"e513bfca-0cd4-4914-9b11-bb153a143c2c\" (UID: \"e513bfca-0cd4-4914-9b11-bb153a143c2c\") "
Nov 01 20:18:43.371423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371384    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume" pod "e513bfca-0cd4-4914-9b11-bb153a143c2c" (UID: "e513bfca-0cd4-4914-9b11-bb153a143c2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:43.371482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371454    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume" pod "e513bfca-0cd4-4914-9b11-bb153a143c2c" (UID: "e513bfca-0cd4-4914-9b11-bb153a143c2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:43.371877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371857    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:43.371982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.371879    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e513bfca-0cd4-4914-9b11-bb153a143c2c-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:43.373212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.373185    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials" pod "e513bfca-0cd4-4914-9b11-bb153a143c2c" (UID: "e513bfca-0cd4-4914-9b11-bb153a143c2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:43.373299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.373250    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb" pod "e513bfca-0cd4-4914-9b11-bb153a143c2c" (UID: "e513bfca-0cd4-4914-9b11-bb153a143c2c"). InnerVolumeSpecName "kube-api-access-vm4hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:43.415525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.415482    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:43.473317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.473283    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:43.473317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.473312    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm4hb\" (UniqueName: \"kubernetes.io/projected/e513bfca-0cd4-4914-9b11-bb153a143c2c-kube-api-access-vm4hb\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:43.770606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.770572    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"
Nov 01 20:18:43.770606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.770603    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld" event={"ID":"b782ebe9-894a-4ca9-bab1-c3bace367c5e","Type":"ContainerDied","Data":"1ad96b33bbb645ab921d0773b6f01e78a8bfdc52f4edf580ee13520de4b1ccc6"}
Nov 01 20:18:43.772280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.772248    6479 generic.go:356] "Generic (PLEG): container finished" podID="e513bfca-0cd4-4914-9b11-bb153a143c2c" containerID="c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5" exitCode=0
Nov 01 20:18:43.772439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.772317    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"
Nov 01 20:18:43.772439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.772382    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk" event={"ID":"e513bfca-0cd4-4914-9b11-bb153a143c2c","Type":"ContainerDied","Data":"c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5"}
Nov 01 20:18:43.772439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.772419    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk" event={"ID":"e513bfca-0cd4-4914-9b11-bb153a143c2c","Type":"ContainerDied","Data":"49202049725d8f8af13d12880768a2fda4f4e07cca162fb0ef9248b98f571f1a"}
Nov 01 20:18:43.772591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.772445    6479 scope.go:122] "RemoveContainer" containerID="c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5"
Nov 01 20:18:43.787917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.787891    6479 scope.go:122] "RemoveContainer" containerID="c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5"
Nov 01 20:18:43.788391 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:43.788359    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5\": container with ID starting with c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5 not found: ID does not exist" containerID="c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5"
Nov 01 20:18:43.788509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.788404    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5"} err="failed to get container status \"c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5\": rpc error: code = NotFound desc = could not find container \"c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5\": container with ID starting with c335c5a85616931d5d1819fa98bb4d536d99c7a15bfff8e57dc0e7e01d324ec5 not found: ID does not exist"
Nov 01 20:18:43.811097 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.811062    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"]
Nov 01 20:18:43.813094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.813048    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:43.818578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.818552    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6d65vvhld"]
Nov 01 20:18:43.840507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.840471    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"]
Nov 01 20:18:43.846923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.846894    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976nzzwk"]
Nov 01 20:18:43.934674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:43.934429    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:44.026824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.023322    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:44.514622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.514583    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"]
Nov 01 20:18:44.516824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.516795    6479 cpu_manager.go:414] "containerMap: removing container" podUID="b782ebe9-894a-4ca9-bab1-c3bace367c5e" containerName="payload-processing"
Nov 01 20:18:44.516824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.516821    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b782ebe9-894a-4ca9-bab1-c3bace367c5e" containerName="payload-processing"
Nov 01 20:18:44.517027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.516881    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e513bfca-0cd4-4914-9b11-bb153a143c2c" containerName="payload-processing"
Nov 01 20:18:44.517027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.516890    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e513bfca-0cd4-4914-9b11-bb153a143c2c" containerName="payload-processing"
Nov 01 20:18:44.517027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.516948    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5a812390-893b-405b-add5-b3f8dd23e4bb" containerName="payload-processing"
Nov 01 20:18:44.517027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.516955    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a812390-893b-405b-add5-b3f8dd23e4bb" containerName="payload-processing"
Nov 01 20:18:44.517275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.517257    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="b782ebe9-894a-4ca9-bab1-c3bace367c5e" containerName="payload-processing"
Nov 01 20:18:44.517344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.517282    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e513bfca-0cd4-4914-9b11-bb153a143c2c" containerName="payload-processing"
Nov 01 20:18:44.517344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.517312    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a812390-893b-405b-add5-b3f8dd23e4bb" containerName="payload-processing"
Nov 01 20:18:44.524284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.524259    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"]
Nov 01 20:18:44.524392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.524348    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.527682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.527636    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1423bbb43273\""
Nov 01 20:18:44.527848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.527780    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 20:18:44.527848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.527841    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 20:18:44.528072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.528049    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 20:18:44.529030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.529014    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 20:18:44.529112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.529020    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-g8lzb\""
Nov 01 20:18:44.623919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.623886    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.624111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.624054    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.624196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.624178    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hfw\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.624287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.624271    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.709210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.709178    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b782ebe9-894a-4ca9-bab1-c3bace367c5e" path="/var/lib/kubelet/pods/b782ebe9-894a-4ca9-bab1-c3bace367c5e/volumes"
Nov 01 20:18:44.709627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.709613    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e513bfca-0cd4-4914-9b11-bb153a143c2c" path="/var/lib/kubelet/pods/e513bfca-0cd4-4914-9b11-bb153a143c2c/volumes"
Nov 01 20:18:44.725324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.725289    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.725454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.725397    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.725515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.725473    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hfw\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.725576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.725538    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.726035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.726009    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.726146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.726110    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.728016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.727998    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.733409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.733383    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hfw\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw\") pod \"payload-processing-external-model-79ddbf8d54-gln7d\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") " pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.839999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.839919    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:44.952546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:44.952509    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:45.123275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:45.123245    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"]
Nov 01 20:18:45.125968 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:18:45.125926    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193e4f98_a267_436c_8783_d0975f229e69.slice/crio-217069e63692508ae5f7984c90cb85b2884bf5c32275905afebd5cdd67bd1efd WatchSource:0}: Error finding container 217069e63692508ae5f7984c90cb85b2884bf5c32275905afebd5cdd67bd1efd: Status 404 returned error can't find the container with id 217069e63692508ae5f7984c90cb85b2884bf5c32275905afebd5cdd67bd1efd
Nov 01 20:18:45.653309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:45.653277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:45.799275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:45.799234    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d" event={"ID":"193e4f98-a267-436c-8783-d0975f229e69","Type":"ContainerStarted","Data":"86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e"}
Nov 01 20:18:45.799489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:45.799314    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:45.799489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:45.799340    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d" event={"ID":"193e4f98-a267-436c-8783-d0975f229e69","Type":"ContainerStarted","Data":"217069e63692508ae5f7984c90cb85b2884bf5c32275905afebd5cdd67bd1efd"}
Nov 01 20:18:45.819137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:45.819084    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d" podStartSLOduration=1.819070957 podStartE2EDuration="1.819070957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:45.815787387 +0000 UTC m=+1171.231419859" watchObservedRunningTime="2026-11-01 20:18:45.819070957 +0000 UTC m=+1171.234703429"
Nov 01 20:18:46.263467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:46.263422    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:46.957596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:46.957554    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:47.404333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:47.404214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:47.504427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:47.504389    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:47.827734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:47.827697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:47.828252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:47.828227    6479 generic.go:356] "Generic (PLEG): container finished" podID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerID="3e93cab6a575d9f260e7e526a0bf2906a437cb74e526fb5910e25a025c233d15" exitCode=0
Nov 01 20:18:47.828368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:47.828299    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" event={"ID":"87573e4d-d137-4a92-aedb-c8e81bdfb470","Type":"ContainerDied","Data":"3e93cab6a575d9f260e7e526a0bf2906a437cb74e526fb5910e25a025c233d15"}
Nov 01 20:18:47.953753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:47.948209    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:48.003849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.003450    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:18:48.011084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.011030    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.011437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.011415    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.011562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.011335    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.011562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.011494    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.011815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.011772    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.012764    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.012864    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.012905    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.012978    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.013031    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.013094    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst\") pod \"87573e4d-d137-4a92-aedb-c8e81bdfb470\" (UID: \"87573e4d-d137-4a92-aedb-c8e81bdfb470\") "
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.013576    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.013589    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.014269    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.014482    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.014583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.014551    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.017725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.016156    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.017725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.016765    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:18:48.017935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.017804    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "kube-api-access-k8jst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:48.018699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.018636    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token" pod "87573e4d-d137-4a92-aedb-c8e81bdfb470" (UID: "87573e4d-d137-4a92-aedb-c8e81bdfb470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:48.115542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115506    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.115542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115538    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.115542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115548    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87573e4d-d137-4a92-aedb-c8e81bdfb470-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.115871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115558    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.115871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115568    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8jst\" (UniqueName: \"kubernetes.io/projected/87573e4d-d137-4a92-aedb-c8e81bdfb470-kube-api-access-k8jst\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.115871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115581    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.115871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.115594    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87573e4d-d137-4a92-aedb-c8e81bdfb470-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.128531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.128499    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:18:48.216154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216069    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216119    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216160    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216183    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216210    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216255    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216339    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216374    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216411    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.216738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216432    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo\") pod \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\" (UID: \"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0\") "
Nov 01 20:18:48.216738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216441    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.216923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216824    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.216923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216836    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:48.216923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.216853    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.217270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.217248    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.217270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.217268    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.217433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.217279    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.217433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.217288    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.217433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.217297    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.219004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.218668    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:18:48.219004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.218772    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:48.219004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.218962    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "kube-api-access-ngnkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:48.219004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.218974    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token" pod "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" (UID: "3aac7bb0-b360-44df-b3a7-7a57d4eee4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:48.318573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.318541    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.318573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.318573    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.318792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.318586    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnkw\" (UniqueName: \"kubernetes.io/projected/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-kube-api-access-ngnkw\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.318792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.318599    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:48.565700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.565606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:48.639174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.639134    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"]
Nov 01 20:18:48.639950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.639917    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy"
Nov 01 20:18:48.639950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.639948    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy"
Nov 01 20:18:48.640130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.639982    6479 cpu_manager.go:414] "containerMap: removing container" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy"
Nov 01 20:18:48.640130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.639990    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy"
Nov 01 20:18:48.640270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.640175    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerName="istio-proxy"
Nov 01 20:18:48.640270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.640192    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" containerName="istio-proxy"
Nov 01 20:18:48.647500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.646206    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:48.647500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.646535    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:48.655397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.650830    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3de00d\""
Nov 01 20:18:48.655875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.655850    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3de00d-dockercfg-qqnjp\""
Nov 01 20:18:48.658604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.658579    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"]
Nov 01 20:18:48.719588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.719557    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"]
Nov 01 20:18:48.724503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.724471    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:48.724995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.724971    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqtl\" (UniqueName: \"kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl\") pod \"payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:48.726194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.725266    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"]
Nov 01 20:18:48.726194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.725483    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.845719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.844273    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") " pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.845719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.844477    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqtl\" (UniqueName: \"kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl\") pod \"payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:48.845719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.844583    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78c29\" (UniqueName: \"kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29\") pod \"payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") " pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.845719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.844732    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:48.875670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.846850    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:48.968678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.968299    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c29\" (UniqueName: \"kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29\") pod \"payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") " pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.968678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.968535    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") " pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.976925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.969676    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") " pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.996675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.991222    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78c29\" (UniqueName: \"kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29\") pod \"payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") " pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.996675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.991566    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:48.996675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.996035    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx" event={"ID":"87573e4d-d137-4a92-aedb-c8e81bdfb470","Type":"ContainerDied","Data":"1b1fcba4a792b951cff55ad61f73b2cca42daf69379ace2e7699c715813c25f6"}
Nov 01 20:18:48.996675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.996102    6479 scope.go:122] "RemoveContainer" containerID="3e93cab6a575d9f260e7e526a0bf2906a437cb74e526fb5910e25a025c233d15"
Nov 01 20:18:48.996675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:48.996276    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"
Nov 01 20:18:49.030679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.030156    6479 generic.go:356] "Generic (PLEG): container finished" podID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" containerID="29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5" exitCode=0
Nov 01 20:18:49.030679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.030254    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" event={"ID":"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0","Type":"ContainerDied","Data":"29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5"}
Nov 01 20:18:49.030679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.030299    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw" event={"ID":"3aac7bb0-b360-44df-b3a7-7a57d4eee4f0","Type":"ContainerDied","Data":"a5716a76bbd5a86b490f007d5216bd6a43b1f15d72bde552f102be8f01fedc25"}
Nov 01 20:18:49.030679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.030394    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"
Nov 01 20:18:49.036666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.036363    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqtl\" (UniqueName: \"kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl\") pod \"payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:49.052438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.052255    6479 scope.go:122] "RemoveContainer" containerID="29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5"
Nov 01 20:18:49.065372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.064495    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"]
Nov 01 20:18:49.070928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.070894    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-7mjcx"]
Nov 01 20:18:49.080359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.080325    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"]
Nov 01 20:18:49.088583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.088561    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-vjwnw"]
Nov 01 20:18:49.091497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.091465    6479 scope.go:122] "RemoveContainer" containerID="29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5"
Nov 01 20:18:49.091939 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:49.091911    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5\": container with ID starting with 29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5 not found: ID does not exist" containerID="29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5"
Nov 01 20:18:49.092012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.091950    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5"} err="failed to get container status \"29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5\": rpc error: code = NotFound desc = could not find container \"29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5\": container with ID starting with 29e8d61a4a875778e1e169ba2c5010485bb021514c41245964e6131fde5713c5 not found: ID does not exist"
Nov 01 20:18:49.262028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.261993    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:49.293879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.293847    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:49.405019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.404938    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"]
Nov 01 20:18:49.662471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.662435    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"]
Nov 01 20:18:49.662766 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:18:49.662743    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9fc178_a81e_4ecc_bd31_55c9ba71a748.slice/crio-b8a580c97a3ead437907b4eae69436bb9a865310282eb9db333a5df2a2429d44 WatchSource:0}: Error finding container b8a580c97a3ead437907b4eae69436bb9a865310282eb9db333a5df2a2429d44: Status 404 returned error can't find the container with id b8a580c97a3ead437907b4eae69436bb9a865310282eb9db333a5df2a2429d44
Nov 01 20:18:49.937834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:49.937749    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:50.051813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.051775    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc" event={"ID":"3a9fc178-a81e-4ecc-bd31-55c9ba71a748","Type":"ContainerStarted","Data":"c0444cbb3b5336701d8d786ef514e1cb7ff2ee9b6311cdc2dc56c5ede99a9bb8"}
Nov 01 20:18:50.052441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.052416    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc" event={"ID":"3a9fc178-a81e-4ecc-bd31-55c9ba71a748","Type":"ContainerStarted","Data":"b8a580c97a3ead437907b4eae69436bb9a865310282eb9db333a5df2a2429d44"}
Nov 01 20:18:50.053874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.053851    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:18:50.065436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.065397    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q" event={"ID":"11dc0799-e0ba-4a65-a916-2d4955ad7cd0","Type":"ContainerStarted","Data":"14b2af8a604265c3c4e940c431575a76a25ab21dfa69c7f9efa9f1254b8ae4b5"}
Nov 01 20:18:50.066837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.066814    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:18:50.067071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.067052    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q" event={"ID":"11dc0799-e0ba-4a65-a916-2d4955ad7cd0","Type":"ContainerStarted","Data":"90680ddefb9414039df33855ae5727902a542f1779a7c739abc36f5e5b3a0491"}
Nov 01 20:18:50.111678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.111585    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc" podStartSLOduration=2.111566531 podStartE2EDuration="2.111566531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:50.090556798 +0000 UTC m=+1175.506189272" watchObservedRunningTime="2026-11-01 20:18:50.111566531 +0000 UTC m=+1175.527199005"
Nov 01 20:18:50.113790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.113732    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q" podStartSLOduration=2.113717346 podStartE2EDuration="2.113717346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:50.109380221 +0000 UTC m=+1175.525012695" watchObservedRunningTime="2026-11-01 20:18:50.113717346 +0000 UTC m=+1175.529349817"
Nov 01 20:18:50.630360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.630332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:50.773042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.773006    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aac7bb0-b360-44df-b3a7-7a57d4eee4f0" path="/var/lib/kubelet/pods/3aac7bb0-b360-44df-b3a7-7a57d4eee4f0/volumes"
Nov 01 20:18:50.774060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:50.774031    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87573e4d-d137-4a92-aedb-c8e81bdfb470" path="/var/lib/kubelet/pods/87573e4d-d137-4a92-aedb-c8e81bdfb470/volumes"
Nov 01 20:18:51.211797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:51.211763    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:51.818787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:51.818744    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:52.500529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:52.500464    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:53.155788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:53.155735    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:18:53.275829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:53.275791    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:54.340272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:54.340231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:54.780023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:54.779988    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:55.539280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:55.539236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:56.513519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.513475    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"]
Nov 01 20:18:56.562068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.546021    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"]
Nov 01 20:18:56.562068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.546158    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.564629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.563221    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:56.564629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.563527    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3de00d-serving-cert\""
Nov 01 20:18:56.777757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.724248    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:56.777757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.757744    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tpw\" (UniqueName: \"kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.777757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.757832    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.777757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.757880    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.777757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.757906    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.939705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.906553    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.939705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.906623    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.939705 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:56.937865    6479 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-serving-cert: secret "maas-api-e2e-api-a-3de00d-serving-cert" not found
Nov 01 20:18:56.939705 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:56.937988    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls podName:5df05e36-ca34-4edd-a3c5-fd338c93f034 nodeName:}" failed. No retries permitted until 2026-11-01 20:18:57.437962828 +0000 UTC m=+1182.853595282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls") pod "maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" (UID: "5df05e36-ca34-4edd-a3c5-fd338c93f034") : secret "maas-api-e2e-api-a-3de00d-serving-cert" not found
Nov 01 20:18:56.958521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.942717    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.958521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.943146    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tpw\" (UniqueName: \"kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.958521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.944345    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:56.958521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:56.945412    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:18:57.075732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.070212    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:57.182675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.140373    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tpw\" (UniqueName: \"kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:57.183570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.183533    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:57.461339 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.458411    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:57.498045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.497528    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\") pod \"maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:57.605978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.605938    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:57.916602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:57.916564    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:58.461078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:58.460903    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"]
Nov 01 20:18:58.510214 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:18:58.510177    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df05e36_ca34_4edd_a3c5_fd338c93f034.slice/crio-307ab7c81f3bb54de38c89473b680c8efceb1e800f984091760da17160414267 WatchSource:0}: Error finding container 307ab7c81f3bb54de38c89473b680c8efceb1e800f984091760da17160414267: Status 404 returned error can't find the container with id 307ab7c81f3bb54de38c89473b680c8efceb1e800f984091760da17160414267
Nov 01 20:18:58.675962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:58.675917    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:59.184898 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:18:59.184856    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:18:59.484196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.484158    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:59.534763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.534355    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" event={"ID":"5df05e36-ca34-4edd-a3c5-fd338c93f034","Type":"ContainerStarted","Data":"f71995da276732fdcd0c1652edfc165971e8a9745d299433c1dfef6bdc90d65f"}
Nov 01 20:18:59.534763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.534428    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" event={"ID":"5df05e36-ca34-4edd-a3c5-fd338c93f034","Type":"ContainerStarted","Data":"307ab7c81f3bb54de38c89473b680c8efceb1e800f984091760da17160414267"}
Nov 01 20:18:59.569670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.566730    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:18:59.585747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.584154    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" podStartSLOduration=2.746532702 podStartE2EDuration="3.58413574s" totalImagesPullingTime="837.603038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:59.578462274 +0000 UTC m=+1184.994094747" watchObservedRunningTime="2026-11-01 20:18:59.58413574 +0000 UTC m=+1184.999768213"
Nov 01 20:18:59.700866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.698239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:18:59.798717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:18:59.798608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:00.277702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.276285    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"]
Nov 01 20:19:00.282951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.282360    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.284395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.284369    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"]
Nov 01 20:19:00.286600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.286575    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-85078fdb41c3\""
Nov 01 20:19:00.411231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.411190    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.411424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.411355    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.411424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.411417    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.411566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.411479    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl442\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.508627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.508562    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:00.516151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.516106    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl442\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.516938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.516911    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.517260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.517240    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.517423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.517405    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.518390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.518368    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.518532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.518454    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.524162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.524139    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl442\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.525592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.525570    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.610734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.608584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:00.614714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.614688    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:00.706290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.703030    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:00.968721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:00.968370    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"]
Nov 01 20:19:01.186079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.186042    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"]
Nov 01 20:19:01.190774 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:19:01.190746    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3361a464_8ed0_4154_ba4a_af26fded6d26.slice/crio-3aba3f2700b38bffffe523f7c67006896c9515c83d911783469566405e06aa0e WatchSource:0}: Error finding container 3aba3f2700b38bffffe523f7c67006896c9515c83d911783469566405e06aa0e: Status 404 returned error can't find the container with id 3aba3f2700b38bffffe523f7c67006896c9515c83d911783469566405e06aa0e
Nov 01 20:19:01.573333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.573147    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv" event={"ID":"3361a464-8ed0-4154-ba4a-af26fded6d26","Type":"ContainerStarted","Data":"85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1"}
Nov 01 20:19:01.573333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.573219    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv" event={"ID":"3361a464-8ed0-4154-ba4a-af26fded6d26","Type":"ContainerStarted","Data":"3aba3f2700b38bffffe523f7c67006896c9515c83d911783469566405e06aa0e"}
Nov 01 20:19:01.580489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.580458    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:01.682675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.681962    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv" podStartSLOduration=1.68194255 podStartE2EDuration="1.68194255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:01.624939405 +0000 UTC m=+1187.040571877" watchObservedRunningTime="2026-11-01 20:19:01.68194255 +0000 UTC m=+1187.097575025"
Nov 01 20:19:01.684669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.683586    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"]
Nov 01 20:19:01.689818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.689791    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.697877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.697849    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fcf37fb5bad9\""
Nov 01 20:19:01.703849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.703815    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"]
Nov 01 20:19:01.758238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.758197    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.759019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.758986    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.759413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.759378    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfrz\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.759503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.759481    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.861792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.861519    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.861792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.861755    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfrz\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.862037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.861817    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.862037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.861906    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.862944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.862916    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.863164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.863138    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.864373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.864348    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:01.870937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:01.870907    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfrz\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz\") pod \"payload-processing-external-model-845fb4dc4b-zrvgp\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:02.038242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.038202    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:02.056305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.055923    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:02.091595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.091560    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:19:02.096607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.096545    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:19:02.487382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.487346    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"]
Nov 01 20:19:02.492968 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:19:02.492933    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddb9671_e23e_45c5_9692_88c0f2c6ba39.slice/crio-947ba883d6f1a628ada64af57a230b617aba12bf8f7c4b9cc0d97dc8c06f8877 WatchSource:0}: Error finding container 947ba883d6f1a628ada64af57a230b617aba12bf8f7c4b9cc0d97dc8c06f8877: Status 404 returned error can't find the container with id 947ba883d6f1a628ada64af57a230b617aba12bf8f7c4b9cc0d97dc8c06f8877
Nov 01 20:19:02.581598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.581555    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp" event={"ID":"5ddb9671-e23e-45c5-9692-88c0f2c6ba39","Type":"ContainerStarted","Data":"947ba883d6f1a628ada64af57a230b617aba12bf8f7c4b9cc0d97dc8c06f8877"}
Nov 01 20:19:02.890777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:02.890621    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:03.634570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:03.634531    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp" event={"ID":"5ddb9671-e23e-45c5-9692-88c0f2c6ba39","Type":"ContainerStarted","Data":"db7bb4980c60c62c033d28d52318acbe520660a054aa1f95abae01f1fcde721c"}
Nov 01 20:19:03.634787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:03.634717    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:03.651573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:03.651517    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp" podStartSLOduration=2.651502298 podStartE2EDuration="2.651502298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:03.650956615 +0000 UTC m=+1189.066589087" watchObservedRunningTime="2026-11-01 20:19:03.651502298 +0000 UTC m=+1189.067134770"
Nov 01 20:19:03.662623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:03.662585    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:04.307312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:04.307269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:04.956144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:04.956109    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:05.559194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:05.559147    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:06.229701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.229659    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:06.579191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.579113    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:19:06.580632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.580604    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv" podUID="3361a464-8ed0-4154-ba4a-af26fded6d26" containerName="payload-processing" containerID="cri-o://85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1" gracePeriod=25
Nov 01 20:19:06.885930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.885897    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:06.969409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.969366    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume\") pod \"3361a464-8ed0-4154-ba4a-af26fded6d26\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") "
Nov 01 20:19:06.969409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.969412    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume\") pod \"3361a464-8ed0-4154-ba4a-af26fded6d26\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") "
Nov 01 20:19:06.969715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.969539    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials\") pod \"3361a464-8ed0-4154-ba4a-af26fded6d26\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") "
Nov 01 20:19:06.969781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.969750    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442\") pod \"3361a464-8ed0-4154-ba4a-af26fded6d26\" (UID: \"3361a464-8ed0-4154-ba4a-af26fded6d26\") "
Nov 01 20:19:06.969893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.969859    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume" pod "3361a464-8ed0-4154-ba4a-af26fded6d26" (UID: "3361a464-8ed0-4154-ba4a-af26fded6d26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:06.970089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.969917    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume" pod "3361a464-8ed0-4154-ba4a-af26fded6d26" (UID: "3361a464-8ed0-4154-ba4a-af26fded6d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:06.970443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.970421    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:06.970528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.970449    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3361a464-8ed0-4154-ba4a-af26fded6d26-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:06.971970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.971949    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials" pod "3361a464-8ed0-4154-ba4a-af26fded6d26" (UID: "3361a464-8ed0-4154-ba4a-af26fded6d26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:06.972048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:06.971979    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442" pod "3361a464-8ed0-4154-ba4a-af26fded6d26" (UID: "3361a464-8ed0-4154-ba4a-af26fded6d26"). InnerVolumeSpecName "kube-api-access-gl442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:07.037486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.037450    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:07.072380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.072345    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl442\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-kube-api-access-gl442\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:07.072380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.072383    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3361a464-8ed0-4154-ba4a-af26fded6d26-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:07.631968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.631932    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:07.764932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.764894    6479 generic.go:356] "Generic (PLEG): container finished" podID="3361a464-8ed0-4154-ba4a-af26fded6d26" containerID="85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1" exitCode=0
Nov 01 20:19:07.765185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.765165    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv" event={"ID":"3361a464-8ed0-4154-ba4a-af26fded6d26","Type":"ContainerDied","Data":"85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1"}
Nov 01 20:19:07.765326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.765310    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv" event={"ID":"3361a464-8ed0-4154-ba4a-af26fded6d26","Type":"ContainerDied","Data":"3aba3f2700b38bffffe523f7c67006896c9515c83d911783469566405e06aa0e"}
Nov 01 20:19:07.765455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.765438    6479 scope.go:122] "RemoveContainer" containerID="85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1"
Nov 01 20:19:07.765730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.765713    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"
Nov 01 20:19:07.790503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.790481    6479 scope.go:122] "RemoveContainer" containerID="85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1"
Nov 01 20:19:07.790821 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:19:07.790799    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1\": container with ID starting with 85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1 not found: ID does not exist" containerID="85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1"
Nov 01 20:19:07.790874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.790828    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1"} err="failed to get container status \"85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1\": rpc error: code = NotFound desc = could not find container \"85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1\": container with ID starting with 85a7bd487d9a26e285c6f3387a3c1a700e69a6197bb38be9cf4bd0affed505f1 not found: ID does not exist"
Nov 01 20:19:07.803344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.803287    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"]
Nov 01 20:19:07.807094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:07.807070    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79rczkv"]
Nov 01 20:19:08.235397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:08.235350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:08.709425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:08.709393    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3361a464-8ed0-4154-ba4a-af26fded6d26" path="/var/lib/kubelet/pods/3361a464-8ed0-4154-ba4a-af26fded6d26/volumes"
Nov 01 20:19:09.031429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:09.031330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:09.755349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:09.755316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:10.345562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:10.345495    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:11.026968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:11.026933    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:11.642345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:11.642309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:12.241223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:12.241184    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:12.489502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:12.489462    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:12.902692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:12.902642    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:13.079482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:13.079430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:13.515967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:13.515922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:13.611758 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:13.611718    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:13.735695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:13.729460    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:13.877920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:13.877825    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"]
Nov 01 20:19:14.239849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:14.239819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:14.644732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:14.644625    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:14.722985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:14.722952    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"]
Nov 01 20:19:15.013479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:15.013442    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:15.654207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:15.654173    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:16.442101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:16.442068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:17.192657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:17.192610    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:17.288882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:17.288848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:17.770996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:17.770940    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:18.493155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:18.493117    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:18.583266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:18.583228    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:18.906295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:18.906205    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz" podUID="3e04449e-185a-43be-9f41-72505bb8a121" containerName="payload-processing" containerID="cri-o://1c3330d4ce9a91d629e20d62668594de368a40034f1018c272f762dc9c3ef4dd" gracePeriod=25
Nov 01 20:19:18.990765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:18.990726    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"]
Nov 01 20:19:18.995560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:18.995524    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3361a464-8ed0-4154-ba4a-af26fded6d26" containerName="payload-processing"
Nov 01 20:19:19.000505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.000479    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3361a464-8ed0-4154-ba4a-af26fded6d26" containerName="payload-processing"
Nov 01 20:19:19.000930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.000910    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3361a464-8ed0-4154-ba4a-af26fded6d26" containerName="payload-processing"
Nov 01 20:19:19.031671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.010860    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.097489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.097449    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qf5v\" (UniqueName: \"kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v\") pod \"payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.097884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.097858    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.105843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.104406    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6afe1a-dockercfg-6qh7v\""
Nov 01 20:19:19.105843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.105111    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6afe1a\""
Nov 01 20:19:19.105843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.010753    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"]
Nov 01 20:19:19.202752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.202586    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.202949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.202767    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qf5v\" (UniqueName: \"kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v\") pod \"payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.222757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.207424    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.226686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.226641    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:19.303670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.289752    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qf5v\" (UniqueName: \"kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v\") pod \"payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.545913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.545810    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:19.729671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:19.728739    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d" podUID="193e4f98-a267-436c-8783-d0975f229e69" containerName="payload-processing" containerID="cri-o://86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e" gracePeriod=25
Nov 01 20:19:20.141295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.141262    6479 generic.go:356] "Generic (PLEG): container finished" podID="3e04449e-185a-43be-9f41-72505bb8a121" containerID="1c3330d4ce9a91d629e20d62668594de368a40034f1018c272f762dc9c3ef4dd" exitCode=0
Nov 01 20:19:20.141557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.141534    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz" event={"ID":"3e04449e-185a-43be-9f41-72505bb8a121","Type":"ContainerDied","Data":"1c3330d4ce9a91d629e20d62668594de368a40034f1018c272f762dc9c3ef4dd"}
Nov 01 20:19:20.306670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.297961    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:20.351687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.350775    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:19:20.443782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.440843    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials\") pod \"3e04449e-185a-43be-9f41-72505bb8a121\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") "
Nov 01 20:19:20.443782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.440990    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume\") pod \"3e04449e-185a-43be-9f41-72505bb8a121\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") "
Nov 01 20:19:20.443782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.441354    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf\") pod \"3e04449e-185a-43be-9f41-72505bb8a121\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") "
Nov 01 20:19:20.443782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.441560    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume\") pod \"3e04449e-185a-43be-9f41-72505bb8a121\" (UID: \"3e04449e-185a-43be-9f41-72505bb8a121\") "
Nov 01 20:19:20.481315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.481265    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume" pod "3e04449e-185a-43be-9f41-72505bb8a121" (UID: "3e04449e-185a-43be-9f41-72505bb8a121"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:20.544388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.544338    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:20.599437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.599367    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf" pod "3e04449e-185a-43be-9f41-72505bb8a121" (UID: "3e04449e-185a-43be-9f41-72505bb8a121"). InnerVolumeSpecName "kube-api-access-g9xdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:20.647671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.636916    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume" pod "3e04449e-185a-43be-9f41-72505bb8a121" (UID: "3e04449e-185a-43be-9f41-72505bb8a121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:20.647671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.641849    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials" pod "3e04449e-185a-43be-9f41-72505bb8a121" (UID: "3e04449e-185a-43be-9f41-72505bb8a121"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:20.647671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.645990    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xdf\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-kube-api-access-g9xdf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:20.647671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.646018    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e04449e-185a-43be-9f41-72505bb8a121-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:20.647671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.646034    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e04449e-185a-43be-9f41-72505bb8a121-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:20.678897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.675699    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"]
Nov 01 20:19:20.800671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.794751    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:19:20.800671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:20.794821    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:21.177864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.176537    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"
Nov 01 20:19:21.177864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.176622    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz" event={"ID":"3e04449e-185a-43be-9f41-72505bb8a121","Type":"ContainerDied","Data":"fd512f78b5ab60773a77eeb6f2c9aa6b411ae37e170a52ad2b5d09bfa9903363"}
Nov 01 20:19:21.177864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.176707    6479 scope.go:122] "RemoveContainer" containerID="1c3330d4ce9a91d629e20d62668594de368a40034f1018c272f762dc9c3ef4dd"
Nov 01 20:19:21.182677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.182514    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf" event={"ID":"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf","Type":"ContainerStarted","Data":"987bf703eb7acd6ff1c6bee02db884cfedb288e06715f718acb26def92887988"}
Nov 01 20:19:21.213121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.213089    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"]
Nov 01 20:19:21.236603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.236529    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745qbrfz"]
Nov 01 20:19:21.538220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:21.538183    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:22.193308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.193271    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf" event={"ID":"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf","Type":"ContainerStarted","Data":"670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f"}
Nov 01 20:19:22.193868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.193353    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:22.224409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.224344    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf" podStartSLOduration=4.224325851 podStartE2EDuration="4.224325851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:22.220742476 +0000 UTC m=+1207.636374952" watchObservedRunningTime="2026-11-01 20:19:22.224325851 +0000 UTC m=+1207.639958331"
Nov 01 20:19:22.396856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.396040    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:22.445630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.445604    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:19:22.606698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.606123    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume\") pod \"193e4f98-a267-436c-8783-d0975f229e69\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") "
Nov 01 20:19:22.606698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.606192    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials\") pod \"193e4f98-a267-436c-8783-d0975f229e69\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") "
Nov 01 20:19:22.606698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.606223    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw\") pod \"193e4f98-a267-436c-8783-d0975f229e69\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") "
Nov 01 20:19:22.606698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.606591    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume\") pod \"193e4f98-a267-436c-8783-d0975f229e69\" (UID: \"193e4f98-a267-436c-8783-d0975f229e69\") "
Nov 01 20:19:22.631669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.618882    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume" pod "193e4f98-a267-436c-8783-d0975f229e69" (UID: "193e4f98-a267-436c-8783-d0975f229e69"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:22.631669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.627932    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume" pod "193e4f98-a267-436c-8783-d0975f229e69" (UID: "193e4f98-a267-436c-8783-d0975f229e69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:22.639231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.639188    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw" pod "193e4f98-a267-436c-8783-d0975f229e69" (UID: "193e4f98-a267-436c-8783-d0975f229e69"). InnerVolumeSpecName "kube-api-access-l5hfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:22.639429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.639319    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials" pod "193e4f98-a267-436c-8783-d0975f229e69" (UID: "193e4f98-a267-436c-8783-d0975f229e69"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:22.709324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.709022    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:22.709324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.709055    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193e4f98-a267-436c-8783-d0975f229e69-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:22.709324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.709066    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:22.709324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.709076    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5hfw\" (UniqueName: \"kubernetes.io/projected/193e4f98-a267-436c-8783-d0975f229e69-kube-api-access-l5hfw\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:22.712809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.712769    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e04449e-185a-43be-9f41-72505bb8a121" path="/var/lib/kubelet/pods/3e04449e-185a-43be-9f41-72505bb8a121/volumes"
Nov 01 20:19:22.747782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:22.747749    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:23.202775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.202739    6479 generic.go:356] "Generic (PLEG): container finished" podID="193e4f98-a267-436c-8783-d0975f229e69" containerID="86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e" exitCode=0
Nov 01 20:19:23.203257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.202797    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"
Nov 01 20:19:23.203257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.202823    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d" event={"ID":"193e4f98-a267-436c-8783-d0975f229e69","Type":"ContainerDied","Data":"86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e"}
Nov 01 20:19:23.203257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.202931    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d" event={"ID":"193e4f98-a267-436c-8783-d0975f229e69","Type":"ContainerDied","Data":"217069e63692508ae5f7984c90cb85b2884bf5c32275905afebd5cdd67bd1efd"}
Nov 01 20:19:23.203257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.202962    6479 scope.go:122] "RemoveContainer" containerID="86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e"
Nov 01 20:19:23.213269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.213240    6479 scope.go:122] "RemoveContainer" containerID="86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e"
Nov 01 20:19:23.213597 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:19:23.213577    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e\": container with ID starting with 86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e not found: ID does not exist" containerID="86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e"
Nov 01 20:19:23.213679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.213605    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e"} err="failed to get container status \"86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e\": rpc error: code = NotFound desc = could not find container \"86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e\": container with ID starting with 86bac3b470aff4d0448ae82310b79bbb179d22c058bc0b29b560b9b7c46a729e not found: ID does not exist"
Nov 01 20:19:23.220839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.220808    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"]
Nov 01 20:19:23.223935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.223913    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79ddbf8d54-gln7d"]
Nov 01 20:19:23.486600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:23.486515    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:24.381212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:24.381178    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:24.711594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:24.711551    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193e4f98-a267-436c-8783-d0975f229e69" path="/var/lib/kubelet/pods/193e4f98-a267-436c-8783-d0975f229e69/volumes"
Nov 01 20:19:24.822387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:24.822326    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:24.922958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:24.922885    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:25.405876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.405841    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"]
Nov 01 20:19:25.406587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.406567    6479 cpu_manager.go:414] "containerMap: removing container" podUID="193e4f98-a267-436c-8783-d0975f229e69" containerName="payload-processing"
Nov 01 20:19:25.406587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.406586    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193e4f98-a267-436c-8783-d0975f229e69" containerName="payload-processing"
Nov 01 20:19:25.406805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.406614    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3e04449e-185a-43be-9f41-72505bb8a121" containerName="payload-processing"
Nov 01 20:19:25.406805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.406620    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e04449e-185a-43be-9f41-72505bb8a121" containerName="payload-processing"
Nov 01 20:19:25.406805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.406731    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="193e4f98-a267-436c-8783-d0975f229e69" containerName="payload-processing"
Nov 01 20:19:25.406805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.406747    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e04449e-185a-43be-9f41-72505bb8a121" containerName="payload-processing"
Nov 01 20:19:25.411889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.411865    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.418524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.418494    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-82b39bcca26d\""
Nov 01 20:19:25.418691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.418587    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-135c2487e510\""
Nov 01 20:19:25.419233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.419212    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"]
Nov 01 20:19:25.459323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.459272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:25.555765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.555724    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.556197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.556169    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzfv\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.556503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.556482    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.560985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.560951    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.662271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.662167    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.662473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.662271    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzfv\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.662473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.662357    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.662473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.662403    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.663218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.663189    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.663218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.663210    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.665540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.665518    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.669914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.669892    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzfv\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv\") pod \"payload-processing-external-model-7854fcbcdd-m6rzs\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") " pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:25.729244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:25.729209    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:26.016353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:26.016322    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"]
Nov 01 20:19:26.017841 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:19:26.017808    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cb7728_be28_4fbd_8695_1b97253b27ae.slice/crio-f0acd43482837959bc88893322132faf9d6d0ba2581472b89e988d8db7373eda WatchSource:0}: Error finding container f0acd43482837959bc88893322132faf9d6d0ba2581472b89e988d8db7373eda: Status 404 returned error can't find the container with id f0acd43482837959bc88893322132faf9d6d0ba2581472b89e988d8db7373eda
Nov 01 20:19:26.254365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:26.254313    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs" event={"ID":"e2cb7728-be28-4fbd-8695-1b97253b27ae","Type":"ContainerStarted","Data":"5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186"}
Nov 01 20:19:26.254594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:26.254381    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs" event={"ID":"e2cb7728-be28-4fbd-8695-1b97253b27ae","Type":"ContainerStarted","Data":"f0acd43482837959bc88893322132faf9d6d0ba2581472b89e988d8db7373eda"}
Nov 01 20:19:26.254594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:26.254445    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:26.271879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:26.271784    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs" podStartSLOduration=1.271769424 podStartE2EDuration="1.271769424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:26.270820588 +0000 UTC m=+1211.686453061" watchObservedRunningTime="2026-11-01 20:19:26.271769424 +0000 UTC m=+1211.687401897"
Nov 01 20:19:26.275092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:26.275065    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:27.021189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:27.021151    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:27.700530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:27.700496    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:28.345350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:28.345317    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:29.045547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.045511    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:29.226434 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:19:29.226396    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:19:29.623847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.623811    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"]
Nov 01 20:19:29.629768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.629743    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.637715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.637682    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6afe1a-serving-cert\""
Nov 01 20:19:29.655757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.655629    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"]
Nov 01 20:19:29.660821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.660787    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.660987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.660843    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwsg\" (UniqueName: \"kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.660987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.660869    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.661201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.661169    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.668017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.667986    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:29.762240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.762199    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.762448 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.762267    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwsg\" (UniqueName: \"kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.762448 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.762299    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.762978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.762599    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.763102 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.763051    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.765165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.765130    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.765359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.765338    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.767853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.767821    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:29.774525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.774488    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwsg\" (UniqueName: \"kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg\") pod \"maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:29.954023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:29.953990    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:30.432471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:30.432428    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"]
Nov 01 20:19:30.565532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:30.565495    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:31.375196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:31.375155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:31.462578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:31.462511    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg" event={"ID":"03ebc8bc-6a2e-42df-9e29-72e05fc5886b","Type":"ContainerStarted","Data":"37a9877cb58c19c7a9201ceb8640883c87f1fd5c915edaec9a438e9e88e54a5c"}
Nov 01 20:19:31.463525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:31.463499    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:31.463745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:31.463722    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg" event={"ID":"03ebc8bc-6a2e-42df-9e29-72e05fc5886b","Type":"ContainerStarted","Data":"c85ee7865dc098a809bb55cf7f84b5368dd00036681ea611af58b758c82c783f"}
Nov 01 20:19:32.052574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:32.052539    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:32.784702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:32.784663    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:33.207053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:33.207020    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:19:33.226231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:33.226166    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg" podStartSLOduration=3.8203102490000003 podStartE2EDuration="4.226147314s" totalImagesPullingTime="405.837065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:31.497483406 +0000 UTC m=+1216.913115880" watchObservedRunningTime="2026-11-01 20:19:33.226147314 +0000 UTC m=+1218.641779791"
Nov 01 20:19:33.547363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:33.547267    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:34.260600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:34.260558    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:34.888838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:34.888791    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:35.564119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:35.564079    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:36.159138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:36.159099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:36.769172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:36.769134    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:37.267061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:37.267017    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:19:37.314977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:37.314933    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"]
Nov 01 20:19:37.477770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:37.477739    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:19:37.551504 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:37.551412    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:38.019976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:38.019941    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:19:38.350454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:38.350374    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:39.236394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:39.236355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:40.044884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:40.044602    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:41.039666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:41.038854    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:42.303276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:42.303232    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:42.318784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:42.318733    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp" podUID="5ddb9671-e23e-45c5-9692-88c0f2c6ba39" containerName="payload-processing" containerID="cri-o://db7bb4980c60c62c033d28d52318acbe520660a054aa1f95abae01f1fcde721c" gracePeriod=25
Nov 01 20:19:42.737786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:42.727176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:43.211304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.211267    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:43.310437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.310398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:43.656232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.656198    6479 generic.go:356] "Generic (PLEG): container finished" podID="5ddb9671-e23e-45c5-9692-88c0f2c6ba39" containerID="db7bb4980c60c62c033d28d52318acbe520660a054aa1f95abae01f1fcde721c" exitCode=0
Nov 01 20:19:43.656405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.656356    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp" event={"ID":"5ddb9671-e23e-45c5-9692-88c0f2c6ba39","Type":"ContainerDied","Data":"db7bb4980c60c62c033d28d52318acbe520660a054aa1f95abae01f1fcde721c"}
Nov 01 20:19:43.851455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.851363    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:43.940033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.939993    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume\") pod \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") "
Nov 01 20:19:43.940219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940068    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials\") pod \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") "
Nov 01 20:19:43.940219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940158    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz\") pod \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") "
Nov 01 20:19:43.940219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940182    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume\") pod \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\" (UID: \"5ddb9671-e23e-45c5-9692-88c0f2c6ba39\") "
Nov 01 20:19:43.940412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940386    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume" pod "5ddb9671-e23e-45c5-9692-88c0f2c6ba39" (UID: "5ddb9671-e23e-45c5-9692-88c0f2c6ba39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:43.940671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940624    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume" pod "5ddb9671-e23e-45c5-9692-88c0f2c6ba39" (UID: "5ddb9671-e23e-45c5-9692-88c0f2c6ba39"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:43.940894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940873    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:43.940969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.940897    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:43.942313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.942291    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials" pod "5ddb9671-e23e-45c5-9692-88c0f2c6ba39" (UID: "5ddb9671-e23e-45c5-9692-88c0f2c6ba39"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:43.942417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:43.942326    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz" pod "5ddb9671-e23e-45c5-9692-88c0f2c6ba39" (UID: "5ddb9671-e23e-45c5-9692-88c0f2c6ba39"). InnerVolumeSpecName "kube-api-access-2nfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:44.052380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.042665    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:44.052380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.042703    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfrz\" (UniqueName: \"kubernetes.io/projected/5ddb9671-e23e-45c5-9692-88c0f2c6ba39-kube-api-access-2nfrz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:44.068507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.068468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:44.666726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.666691    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"
Nov 01 20:19:44.667227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.666690    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp" event={"ID":"5ddb9671-e23e-45c5-9692-88c0f2c6ba39","Type":"ContainerDied","Data":"947ba883d6f1a628ada64af57a230b617aba12bf8f7c4b9cc0d97dc8c06f8877"}
Nov 01 20:19:44.667227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.666865    6479 scope.go:122] "RemoveContainer" containerID="db7bb4980c60c62c033d28d52318acbe520660a054aa1f95abae01f1fcde721c"
Nov 01 20:19:44.693209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.693165    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"]
Nov 01 20:19:44.696329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.696293    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-zrvgp"]
Nov 01 20:19:44.709989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.709960    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddb9671-e23e-45c5-9692-88c0f2c6ba39" path="/var/lib/kubelet/pods/5ddb9671-e23e-45c5-9692-88c0f2c6ba39/volumes"
Nov 01 20:19:44.776616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:44.776558    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:45.517632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:45.517598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:46.000340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.000303    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-d5327f"]
Nov 01 20:19:46.001053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.001037    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5ddb9671-e23e-45c5-9692-88c0f2c6ba39" containerName="payload-processing"
Nov 01 20:19:46.001053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.001053    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddb9671-e23e-45c5-9692-88c0f2c6ba39" containerName="payload-processing"
Nov 01 20:19:46.001175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.001166    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddb9671-e23e-45c5-9692-88c0f2c6ba39" containerName="payload-processing"
Nov 01 20:19:46.006149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.006124    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:46.009853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.009828    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-d5327f"]
Nov 01 20:19:46.010210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.010185    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\""
Nov 01 20:19:46.085465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.085424    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qcc\" (UniqueName: \"kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc\") pod \"test-curl-7417-d5327f\" (UID: \"a75b0e59-9e80-400c-af63-5050b8832ae0\") " pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:46.179160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.179117    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:46.187027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.186985    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qcc\" (UniqueName: \"kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc\") pod \"test-curl-7417-d5327f\" (UID: \"a75b0e59-9e80-400c-af63-5050b8832ae0\") " pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:46.194896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.194864    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qcc\" (UniqueName: \"kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc\") pod \"test-curl-7417-d5327f\" (UID: \"a75b0e59-9e80-400c-af63-5050b8832ae0\") " pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:46.307298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.307198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:46.326667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.326288    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:46.644503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.644473    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-d5327f"]
Nov 01 20:19:46.645236 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:19:46.645207    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda75b0e59_9e80_400c_af63_5050b8832ae0.slice/crio-d007cb12fb33275ac7cdd3e7f4baf1a4577cbe73abefe398f79b3bce291268b7 WatchSource:0}: Error finding container d007cb12fb33275ac7cdd3e7f4baf1a4577cbe73abefe398f79b3bce291268b7: Status 404 returned error can't find the container with id d007cb12fb33275ac7cdd3e7f4baf1a4577cbe73abefe398f79b3bce291268b7
Nov 01 20:19:46.689642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.689598    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-d5327f" event={"ID":"a75b0e59-9e80-400c-af63-5050b8832ae0","Type":"ContainerStarted","Data":"d007cb12fb33275ac7cdd3e7f4baf1a4577cbe73abefe398f79b3bce291268b7"}
Nov 01 20:19:46.962495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:46.962452    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:47.582621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:47.582577    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:48.320523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:48.320486    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:49.051473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:49.051437    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:49.642781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:49.642746    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:49.809447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:49.809400    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-d5327f" event={"ID":"a75b0e59-9e80-400c-af63-5050b8832ae0","Type":"ContainerStarted","Data":"6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72"}
Nov 01 20:19:49.826972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:49.826913    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-d5327f" podStartSLOduration=2.375641114 podStartE2EDuration="4.82689814s" totalImagesPullingTime="2.451257026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:49.823256311 +0000 UTC m=+1235.238888791" watchObservedRunningTime="2026-11-01 20:19:49.82689814 +0000 UTC m=+1235.242530613"
Nov 01 20:19:50.441439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:50.441405    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:50.468336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:50.468288    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-d5327f"]
Nov 01 20:19:50.472547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:50.472509    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-d5327f"]
Nov 01 20:19:51.137596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:51.137559    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:51.816456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:51.816416    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"]
Nov 01 20:19:51.824373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:51.824330    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-d5327f" podUID="a75b0e59-9e80-400c-af63-5050b8832ae0" containerName="test-curl-7417-d5327f" containerID="cri-o://6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72" gracePeriod=2
Nov 01 20:19:51.832615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:51.832582    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"]
Nov 01 20:19:51.889762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:51.889725    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:51.980602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:51.980567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:52.178548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:52.178461    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:52.369005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:52.368927    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:53.139858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:53.139824    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:53.740659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:53.740622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:54.010231 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:19:54.010195    6479 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda75b0e59_9e80_400c_af63_5050b8832ae0.slice/crio-conmon-6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 20:19:54.118204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.118178    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:54.188831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.188800    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc\" (UniqueName: \"kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc\") pod \"a75b0e59-9e80-400c-af63-5050b8832ae0\" (UID: \"a75b0e59-9e80-400c-af63-5050b8832ae0\") "
Nov 01 20:19:54.191732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.191696    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc" pod "a75b0e59-9e80-400c-af63-5050b8832ae0" (UID: "a75b0e59-9e80-400c-af63-5050b8832ae0"). InnerVolumeSpecName "kube-api-access-x9qcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:54.290839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.290753    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9qcc\" (UniqueName: \"kubernetes.io/projected/a75b0e59-9e80-400c-af63-5050b8832ae0-kube-api-access-x9qcc\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:54.358628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.358593    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:54.722962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.722922    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75b0e59-9e80-400c-af63-5050b8832ae0" path="/var/lib/kubelet/pods/a75b0e59-9e80-400c-af63-5050b8832ae0/volumes"
Nov 01 20:19:54.860383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.860332    6479 generic.go:356] "Generic (PLEG): container finished" podID="a75b0e59-9e80-400c-af63-5050b8832ae0" containerID="6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72" exitCode=137
Nov 01 20:19:54.860568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.860409    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-d5327f"
Nov 01 20:19:54.860568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.860446    6479 scope.go:122] "RemoveContainer" containerID="6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72"
Nov 01 20:19:54.873573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.873543    6479 scope.go:122] "RemoveContainer" containerID="6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72"
Nov 01 20:19:54.873983 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:19:54.873956    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72\": container with ID starting with 6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72 not found: ID does not exist" containerID="6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72"
Nov 01 20:19:54.874070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:54.873994    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72"} err="failed to get container status \"6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72\": rpc error: code = NotFound desc = could not find container \"6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72\": container with ID starting with 6cb64f9afd594d0495df8bf0b30aca63e877b7f3b71da37477067760362a3e72 not found: ID does not exist"
Nov 01 20:19:55.147591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:55.147490    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:55.668029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:55.667989    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"]
Nov 01 20:19:55.668491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:55.668235    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" podUID="5df05e36-ca34-4edd-a3c5-fd338c93f034" containerName="maas-api" containerID="cri-o://f71995da276732fdcd0c1652edfc165971e8a9745d299433c1dfef6bdc90d65f" gracePeriod=30
Nov 01 20:19:55.712908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:55.712870    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:56.071571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.070890    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:56.109301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.109270    6479 generic.go:356] "Generic (PLEG): container finished" podID="5df05e36-ca34-4edd-a3c5-fd338c93f034" containerID="f71995da276732fdcd0c1652edfc165971e8a9745d299433c1dfef6bdc90d65f" exitCode=0
Nov 01 20:19:56.109695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.109632    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" event={"ID":"5df05e36-ca34-4edd-a3c5-fd338c93f034","Type":"ContainerDied","Data":"f71995da276732fdcd0c1652edfc165971e8a9745d299433c1dfef6bdc90d65f"}
Nov 01 20:19:56.286030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.285995    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:19:56.434119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.434024    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca\") pod \"5df05e36-ca34-4edd-a3c5-fd338c93f034\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") "
Nov 01 20:19:56.434277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.434174    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs\") pod \"5df05e36-ca34-4edd-a3c5-fd338c93f034\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") "
Nov 01 20:19:56.434277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.434237    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\") pod \"5df05e36-ca34-4edd-a3c5-fd338c93f034\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") "
Nov 01 20:19:56.434277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.434260    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw\" (UniqueName: \"kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw\") pod \"5df05e36-ca34-4edd-a3c5-fd338c93f034\" (UID: \"5df05e36-ca34-4edd-a3c5-fd338c93f034\") "
Nov 01 20:19:56.434415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.434369    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca" pod "5df05e36-ca34-4edd-a3c5-fd338c93f034" (UID: "5df05e36-ca34-4edd-a3c5-fd338c93f034"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:56.434750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.434728    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df05e36-ca34-4edd-a3c5-fd338c93f034-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:56.436499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.436474    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls" pod "5df05e36-ca34-4edd-a3c5-fd338c93f034" (UID: "5df05e36-ca34-4edd-a3c5-fd338c93f034"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:19:56.436499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.436477    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw" pod "5df05e36-ca34-4edd-a3c5-fd338c93f034" (UID: "5df05e36-ca34-4edd-a3c5-fd338c93f034"). InnerVolumeSpecName "kube-api-access-t7tpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:56.436672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.436518    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs" pod "5df05e36-ca34-4edd-a3c5-fd338c93f034" (UID: "5df05e36-ca34-4edd-a3c5-fd338c93f034"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:19:56.536285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.536233    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:56.536285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.536280    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df05e36-ca34-4edd-a3c5-fd338c93f034-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:56.536513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.536295    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7tpw\" (UniqueName: \"kubernetes.io/projected/5df05e36-ca34-4edd-a3c5-fd338c93f034-kube-api-access-t7tpw\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:56.683548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.683511    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:56.826013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.824728    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q" podUID="11dc0799-e0ba-4a65-a916-2d4955ad7cd0" containerName="payload-processing" containerID="cri-o://14b2af8a604265c3c4e940c431575a76a25ab21dfa69c7f9efa9f1254b8ae4b5" gracePeriod=25
Nov 01 20:19:56.848687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:56.833685    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc" podUID="3a9fc178-a81e-4ecc-bd31-55c9ba71a748" containerName="payload-pre-processing" containerID="cri-o://c0444cbb3b5336701d8d786ef514e1cb7ff2ee9b6311cdc2dc56c5ede99a9bb8" gracePeriod=25
Nov 01 20:19:57.120898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.120672    6479 generic.go:356] "Generic (PLEG): container finished" podID="11dc0799-e0ba-4a65-a916-2d4955ad7cd0" containerID="14b2af8a604265c3c4e940c431575a76a25ab21dfa69c7f9efa9f1254b8ae4b5" exitCode=0
Nov 01 20:19:57.120898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.120754    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q" event={"ID":"11dc0799-e0ba-4a65-a916-2d4955ad7cd0","Type":"ContainerDied","Data":"14b2af8a604265c3c4e940c431575a76a25ab21dfa69c7f9efa9f1254b8ae4b5"}
Nov 01 20:19:57.122382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.122351    6479 generic.go:356] "Generic (PLEG): container finished" podID="3a9fc178-a81e-4ecc-bd31-55c9ba71a748" containerID="c0444cbb3b5336701d8d786ef514e1cb7ff2ee9b6311cdc2dc56c5ede99a9bb8" exitCode=0
Nov 01 20:19:57.122504 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.122423    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc" event={"ID":"3a9fc178-a81e-4ecc-bd31-55c9ba71a748","Type":"ContainerDied","Data":"c0444cbb3b5336701d8d786ef514e1cb7ff2ee9b6311cdc2dc56c5ede99a9bb8"}
Nov 01 20:19:57.122583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.122542    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc" event={"ID":"3a9fc178-a81e-4ecc-bd31-55c9ba71a748","Type":"ContainerDied","Data":"b8a580c97a3ead437907b4eae69436bb9a865310282eb9db333a5df2a2429d44"}
Nov 01 20:19:57.122660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.122596    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a580c97a3ead437907b4eae69436bb9a865310282eb9db333a5df2a2429d44"
Nov 01 20:19:57.124114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.124096    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"
Nov 01 20:19:57.124221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.124094    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l" event={"ID":"5df05e36-ca34-4edd-a3c5-fd338c93f034","Type":"ContainerDied","Data":"307ab7c81f3bb54de38c89473b680c8efceb1e800f984091760da17160414267"}
Nov 01 20:19:57.124221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.124215    6479 scope.go:122] "RemoveContainer" containerID="f71995da276732fdcd0c1652edfc165971e8a9745d299433c1dfef6bdc90d65f"
Nov 01 20:19:57.156722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.156689    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:19:57.158964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.158940    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"]
Nov 01 20:19:57.163928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.163896    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3de00d-6df6cccd94-l6n4l"]
Nov 01 20:19:57.191352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.191317    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:19:57.249177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.249089    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl\" (UniqueName: \"kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl\") pod \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") "
Nov 01 20:19:57.249344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.249196    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume\") pod \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\" (UID: \"3a9fc178-a81e-4ecc-bd31-55c9ba71a748\") "
Nov 01 20:19:57.249593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.249568    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume" pod "3a9fc178-a81e-4ecc-bd31-55c9ba71a748" (UID: "3a9fc178-a81e-4ecc-bd31-55c9ba71a748"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:57.249768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.249746    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:57.251205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.251180    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl" pod "3a9fc178-a81e-4ecc-bd31-55c9ba71a748" (UID: "3a9fc178-a81e-4ecc-bd31-55c9ba71a748"). InnerVolumeSpecName "kube-api-access-zlqtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:57.350289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.350250    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29\" (UniqueName: \"kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29\") pod \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") "
Nov 01 20:19:57.350462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.350364    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume\") pod \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\" (UID: \"11dc0799-e0ba-4a65-a916-2d4955ad7cd0\") "
Nov 01 20:19:57.350767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.350736    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume" pod "11dc0799-e0ba-4a65-a916-2d4955ad7cd0" (UID: "11dc0799-e0ba-4a65-a916-2d4955ad7cd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:57.350881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.350802    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqtl\" (UniqueName: \"kubernetes.io/projected/3a9fc178-a81e-4ecc-bd31-55c9ba71a748-kube-api-access-zlqtl\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:57.352450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.352421    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29" pod "11dc0799-e0ba-4a65-a916-2d4955ad7cd0" (UID: "11dc0799-e0ba-4a65-a916-2d4955ad7cd0"). InnerVolumeSpecName "kube-api-access-78c29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:57.452399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.452355    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:57.452399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.452391    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78c29\" (UniqueName: \"kubernetes.io/projected/11dc0799-e0ba-4a65-a916-2d4955ad7cd0-kube-api-access-78c29\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:57.569623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.569527    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:57.736581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.736546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:57.898070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:57.898008    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:58.141742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.141658    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"
Nov 01 20:19:58.141742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.141677    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q" event={"ID":"11dc0799-e0ba-4a65-a916-2d4955ad7cd0","Type":"ContainerDied","Data":"90680ddefb9414039df33855ae5727902a542f1779a7c739abc36f5e5b3a0491"}
Nov 01 20:19:58.141988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.141772    6479 scope.go:122] "RemoveContainer" containerID="14b2af8a604265c3c4e940c431575a76a25ab21dfa69c7f9efa9f1254b8ae4b5"
Nov 01 20:19:58.141988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.141891    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"
Nov 01 20:19:58.169772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.169661    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"]
Nov 01 20:19:58.175235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.175204    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3de00d-67d695865f-zwnlc"]
Nov 01 20:19:58.189597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.189565    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"]
Nov 01 20:19:58.192511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.192489    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3de00d-67d9bf7fcf-b4d8q"]
Nov 01 20:19:58.328763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.328721    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:58.547291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.547258    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:58.661572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.661535    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:58.710249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.710216    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dc0799-e0ba-4a65-a916-2d4955ad7cd0" path="/var/lib/kubelet/pods/11dc0799-e0ba-4a65-a916-2d4955ad7cd0/volumes"
Nov 01 20:19:58.710728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.710711    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9fc178-a81e-4ecc-bd31-55c9ba71a748" path="/var/lib/kubelet/pods/3a9fc178-a81e-4ecc-bd31-55c9ba71a748/volumes"
Nov 01 20:19:58.711152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:58.711139    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df05e36-ca34-4edd-a3c5-fd338c93f034" path="/var/lib/kubelet/pods/5df05e36-ca34-4edd-a3c5-fd338c93f034/volumes"
Nov 01 20:19:59.175008 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:19:59.174954    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:19:59.381078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:59.381041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:19:59.483703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:19:59.483665    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:00.163628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:00.163587    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:00.824881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:00.824849    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:01.556463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:01.556419    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:02.081347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.081310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:02.203708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.195599    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.240704    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"]
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241811    6479 cpu_manager.go:414] "containerMap: removing container" podUID="11dc0799-e0ba-4a65-a916-2d4955ad7cd0" containerName="payload-processing"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241836    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dc0799-e0ba-4a65-a916-2d4955ad7cd0" containerName="payload-processing"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241863    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3a9fc178-a81e-4ecc-bd31-55c9ba71a748" containerName="payload-pre-processing"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241870    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9fc178-a81e-4ecc-bd31-55c9ba71a748" containerName="payload-pre-processing"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241924    6479 cpu_manager.go:414] "containerMap: removing container" podUID="a75b0e59-9e80-400c-af63-5050b8832ae0" containerName="test-curl-7417-d5327f"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241932    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75b0e59-9e80-400c-af63-5050b8832ae0" containerName="test-curl-7417-d5327f"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241960    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5df05e36-ca34-4edd-a3c5-fd338c93f034" containerName="maas-api"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.241968    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df05e36-ca34-4edd-a3c5-fd338c93f034" containerName="maas-api"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.242139    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df05e36-ca34-4edd-a3c5-fd338c93f034" containerName="maas-api"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.242170    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dc0799-e0ba-4a65-a916-2d4955ad7cd0" containerName="payload-processing"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.242182    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9fc178-a81e-4ecc-bd31-55c9ba71a748" containerName="payload-pre-processing"
Nov 01 20:20:02.248671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.242200    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75b0e59-9e80-400c-af63-5050b8832ae0" containerName="test-curl-7417-d5327f"
Nov 01 20:20:02.416763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.414723    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"]
Nov 01 20:20:02.416763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.414837    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.519350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.519310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:02.584887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.584840    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.585236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.585210    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.585431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.585409    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.585631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.585613    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9b29\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.728698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.727621    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.728698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.727717    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.728698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.727751    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.728698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.727809    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9b29\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.736699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.729186    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.736699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.730206    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.739956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.739925    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:02.797425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:02.797362    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9b29\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29\") pod \"payload-processing-external-model-845fb4dc4b-s2wjz\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") " pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:03.034858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:03.034771    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:03.329400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:03.329368    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"]
Nov 01 20:20:03.332203 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:03.332173    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620e95cf_d32f_4ce8_b614_14a27fee162e.slice/crio-7eaf172114d2e3d4f95af8495456947a327f4c9aa002d1d04ae95b6abccce35b WatchSource:0}: Error finding container 7eaf172114d2e3d4f95af8495456947a327f4c9aa002d1d04ae95b6abccce35b: Status 404 returned error can't find the container with id 7eaf172114d2e3d4f95af8495456947a327f4c9aa002d1d04ae95b6abccce35b
Nov 01 20:20:03.377422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:03.377386    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:04.190968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.190922    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz" event={"ID":"620e95cf-d32f-4ce8-b614-14a27fee162e","Type":"ContainerStarted","Data":"087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8"}
Nov 01 20:20:04.191162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.190990    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz" event={"ID":"620e95cf-d32f-4ce8-b614-14a27fee162e","Type":"ContainerStarted","Data":"7eaf172114d2e3d4f95af8495456947a327f4c9aa002d1d04ae95b6abccce35b"}
Nov 01 20:20:04.191240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.191208    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:04.212853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.212785    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz" podStartSLOduration=2.212764341 podStartE2EDuration="2.212764341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:04.20687925 +0000 UTC m=+1249.622511723" watchObservedRunningTime="2026-11-01 20:20:04.212764341 +0000 UTC m=+1249.628396815"
Nov 01 20:20:04.444526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.444498    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:04.768933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.768833    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"]
Nov 01 20:20:04.819276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.819191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:04.919009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:04.918973    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:05.435765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:05.435728    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:05.631118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:05.631084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:06.076222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.076181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:06.403574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.403473    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:06.503808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.503750    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:06.920089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.920058    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"]
Nov 01 20:20:06.925035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.925014    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:06.929775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.929747    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\""
Nov 01 20:20:06.932553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.932519    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\""
Nov 01 20:20:06.932702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.932595    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\""
Nov 01 20:20:06.932702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.932690    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\""
Nov 01 20:20:06.932878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.932789    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\""
Nov 01 20:20:06.932878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.932849    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-b89fbe06e26a\""
Nov 01 20:20:06.933372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.933348    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-85078fdb41c3\""
Nov 01 20:20:06.933547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.933529    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-8876b9054304\""
Nov 01 20:20:06.933618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.933586    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-3f909928dd00\""
Nov 01 20:20:06.933690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.933663    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\""
Nov 01 20:20:06.934004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:06.933988    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"]
Nov 01 20:20:07.077935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.077895    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.078090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.078029    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.078247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.078217    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zvj\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.078329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.078313    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.180189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.180092    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.180366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.180195    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.180366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.180338    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zvj\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.180508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.180391    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.181250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.181219    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.181250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.181233    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.183177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.183152    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.187939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.187903    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zvj\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.242480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.242440    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:07.319836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.319763    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:07.540709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.540679    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"]
Nov 01 20:20:07.542101 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:07.542073    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8193030a_964c_4f08_9d82_9feeb0085864.slice/crio-9ebe338704f365c1c76c133e63d0ca38375beffdb9b7e3459c6583c5462cbceb WatchSource:0}: Error finding container 9ebe338704f365c1c76c133e63d0ca38375beffdb9b7e3459c6583c5462cbceb: Status 404 returned error can't find the container with id 9ebe338704f365c1c76c133e63d0ca38375beffdb9b7e3459c6583c5462cbceb
Nov 01 20:20:07.968728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:07.968687    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:08.274437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.274328    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk" event={"ID":"8193030a-964c-4f08-9d82-9feeb0085864","Type":"ContainerStarted","Data":"456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2"}
Nov 01 20:20:08.274437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.274402    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk" event={"ID":"8193030a-964c-4f08-9d82-9feeb0085864","Type":"ContainerStarted","Data":"9ebe338704f365c1c76c133e63d0ca38375beffdb9b7e3459c6583c5462cbceb"}
Nov 01 20:20:08.274437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.274433    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:08.293218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.293158    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk" podStartSLOduration=2.293143125 podStartE2EDuration="2.293143125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:08.290910781 +0000 UTC m=+1253.706543256" watchObservedRunningTime="2026-11-01 20:20:08.293143125 +0000 UTC m=+1253.708775598"
Nov 01 20:20:08.484545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.484503    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:08.586840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.586748    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:08.696635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:08.692628    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:09.333006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:09.332968    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:09.770272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:09.770213    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf" podUID="0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" containerName="payload-pre-processing" containerID="cri-o://670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f" gracePeriod=25
Nov 01 20:20:09.938238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:09.938197    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:10.152678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.151970    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:20:10.266355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.266326    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v\" (UniqueName: \"kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v\") pod \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") "
Nov 01 20:20:10.266526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.266472    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume\") pod \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\" (UID: \"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf\") "
Nov 01 20:20:10.266862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.266839    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume" pod "0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" (UID: "0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:10.267074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.267054    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:10.268686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.268661    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v" pod "0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" (UID: "0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf"). InnerVolumeSpecName "kube-api-access-7qf5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:10.292406 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.292372    6479 generic.go:356] "Generic (PLEG): container finished" podID="0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" containerID="670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f" exitCode=0
Nov 01 20:20:10.292578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.292432    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"
Nov 01 20:20:10.292578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.292464    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf" event={"ID":"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf","Type":"ContainerDied","Data":"670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f"}
Nov 01 20:20:10.292578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.292532    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf" event={"ID":"0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf","Type":"ContainerDied","Data":"987bf703eb7acd6ff1c6bee02db884cfedb288e06715f718acb26def92887988"}
Nov 01 20:20:10.292578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.292563    6479 scope.go:122] "RemoveContainer" containerID="670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f"
Nov 01 20:20:10.305815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.305796    6479 scope.go:122] "RemoveContainer" containerID="670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f"
Nov 01 20:20:10.306144 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:10.306123    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f\": container with ID starting with 670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f not found: ID does not exist" containerID="670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f"
Nov 01 20:20:10.306203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.306153    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f"} err="failed to get container status \"670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f\": rpc error: code = NotFound desc = could not find container \"670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f\": container with ID starting with 670d1b3bed9fdba4e2ae7fd0ac77d8fd988e495958cab654ce09d0a40fac706f not found: ID does not exist"
Nov 01 20:20:10.315869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.315834    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"]
Nov 01 20:20:10.318906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.318884    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6afe1a-797bfdbc-n58sf"]
Nov 01 20:20:10.368396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.368314    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qf5v\" (UniqueName: \"kubernetes.io/projected/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf-kube-api-access-7qf5v\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:10.540698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.540633    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:10.711188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:10.711149    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" path="/var/lib/kubelet/pods/0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf/volumes"
Nov 01 20:20:11.025010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:11.024924    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"]
Nov 01 20:20:11.027955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:11.027927    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddffb270-26h7d"]
Nov 01 20:20:11.199015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:11.198980    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:11.611800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:11.611762    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:11.862185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:11.862098    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:12.597053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:12.597018    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:12.769673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:12.758943    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee24b5b6-f3d9-4154-8e64-24eaf58f658b" path="/var/lib/kubelet/pods/ee24b5b6-f3d9-4154-8e64-24eaf58f658b/volumes"
Nov 01 20:20:12.822797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:12.822769    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:13.570305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:13.570265    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:13.884905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:13.884821    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"]
Nov 01 20:20:13.888902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:13.888874    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:13.890265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:13.890238    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"]
Nov 01 20:20:13.895326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:13.895294    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:13.995050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:13.995018    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:14.654195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:14.654145    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:15.201135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.200991    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"]
Nov 01 20:20:15.201729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.201470    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg" podUID="03ebc8bc-6a2e-42df-9e29-72e05fc5886b" containerName="maas-api" containerID="cri-o://37a9877cb58c19c7a9201ceb8640883c87f1fd5c915edaec9a438e9e88e54a5c" gracePeriod=30
Nov 01 20:20:15.242591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.242552    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:15.342800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.342772    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:15.603454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.602112    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:15.604101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.604077    6479 generic.go:356] "Generic (PLEG): container finished" podID="03ebc8bc-6a2e-42df-9e29-72e05fc5886b" containerID="37a9877cb58c19c7a9201ceb8640883c87f1fd5c915edaec9a438e9e88e54a5c" exitCode=0
Nov 01 20:20:15.604353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.604148    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg" event={"ID":"03ebc8bc-6a2e-42df-9e29-72e05fc5886b","Type":"ContainerDied","Data":"37a9877cb58c19c7a9201ceb8640883c87f1fd5c915edaec9a438e9e88e54a5c"}
Nov 01 20:20:15.829980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.829949    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:15.878750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.871853    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:20:15.906679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.906212    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls\") pod \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") "
Nov 01 20:20:15.906679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.906330    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg\" (UniqueName: \"kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg\") pod \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") "
Nov 01 20:20:15.906679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.906414    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca\") pod \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") "
Nov 01 20:20:15.906679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.906537    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs\") pod \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\" (UID: \"03ebc8bc-6a2e-42df-9e29-72e05fc5886b\") "
Nov 01 20:20:15.920681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.912824    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca" pod "03ebc8bc-6a2e-42df-9e29-72e05fc5886b" (UID: "03ebc8bc-6a2e-42df-9e29-72e05fc5886b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:15.924475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.921597    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg" pod "03ebc8bc-6a2e-42df-9e29-72e05fc5886b" (UID: "03ebc8bc-6a2e-42df-9e29-72e05fc5886b"). InnerVolumeSpecName "kube-api-access-lwwsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:15.973628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.972184    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs" pod "03ebc8bc-6a2e-42df-9e29-72e05fc5886b" (UID: "03ebc8bc-6a2e-42df-9e29-72e05fc5886b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:15.973628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:15.972265    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls" pod "03ebc8bc-6a2e-42df-9e29-72e05fc5886b" (UID: "03ebc8bc-6a2e-42df-9e29-72e05fc5886b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:16.035768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.011966    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwsg\" (UniqueName: \"kubernetes.io/projected/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-kube-api-access-lwwsg\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:16.035768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.012004    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:16.035768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.012022    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:16.035768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.012035    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ebc8bc-6a2e-42df-9e29-72e05fc5886b-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:16.040400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.040312    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:16.446748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.446709    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:16.616142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.616111    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"
Nov 01 20:20:16.616338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.616143    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg" event={"ID":"03ebc8bc-6a2e-42df-9e29-72e05fc5886b","Type":"ContainerDied","Data":"c85ee7865dc098a809bb55cf7f84b5368dd00036681ea611af58b758c82c783f"}
Nov 01 20:20:16.616338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.616213    6479 scope.go:122] "RemoveContainer" containerID="37a9877cb58c19c7a9201ceb8640883c87f1fd5c915edaec9a438e9e88e54a5c"
Nov 01 20:20:16.642430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.642394    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"]
Nov 01 20:20:16.645424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.645402    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6afe1a-5f9dd4fd57-f6jgg"]
Nov 01 20:20:16.709879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:16.709799    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ebc8bc-6a2e-42df-9e29-72e05fc5886b" path="/var/lib/kubelet/pods/03ebc8bc-6a2e-42df-9e29-72e05fc5886b/volumes"
Nov 01 20:20:17.046989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:17.046894    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:17.675706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:17.675666    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:17.776273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:17.776241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:17.923906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:17.923867    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:18.018080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:18.018045    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:18.150502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:18.150467    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:18.499940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:18.499407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:18.612668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:18.609173    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:18.885873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:18.885785    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz" podUID="620e95cf-d32f-4ce8-b614-14a27fee162e" containerName="payload-processing" containerID="cri-o://087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8" gracePeriod=25
Nov 01 20:20:18.890946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:18.890909    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs" podUID="e2cb7728-be28-4fbd-8695-1b97253b27ae" containerName="payload-processing" containerID="cri-o://5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186" gracePeriod=25
Nov 01 20:20:19.140692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.140559    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:19.300675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.299941    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:20:19.315761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.315718    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:19.319656 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.319034    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:19.382713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.378986    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:20:19.408313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.408239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:19.417380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.417341    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials\") pod \"620e95cf-d32f-4ce8-b614-14a27fee162e\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") "
Nov 01 20:20:19.417549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.417437    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume\") pod \"620e95cf-d32f-4ce8-b614-14a27fee162e\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") "
Nov 01 20:20:19.417549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.417512    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume\") pod \"620e95cf-d32f-4ce8-b614-14a27fee162e\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") "
Nov 01 20:20:19.417714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.417691    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29\") pod \"620e95cf-d32f-4ce8-b614-14a27fee162e\" (UID: \"620e95cf-d32f-4ce8-b614-14a27fee162e\") "
Nov 01 20:20:19.418063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.418035    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume" pod "620e95cf-d32f-4ce8-b614-14a27fee162e" (UID: "620e95cf-d32f-4ce8-b614-14a27fee162e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:19.418207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.418054    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume" pod "620e95cf-d32f-4ce8-b614-14a27fee162e" (UID: "620e95cf-d32f-4ce8-b614-14a27fee162e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:19.418512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.418487    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.418622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.418516    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e95cf-d32f-4ce8-b614-14a27fee162e-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.420449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.420425    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29" pod "620e95cf-d32f-4ce8-b614-14a27fee162e" (UID: "620e95cf-d32f-4ce8-b614-14a27fee162e"). InnerVolumeSpecName "kube-api-access-v9b29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:19.420565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.420459    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials" pod "620e95cf-d32f-4ce8-b614-14a27fee162e" (UID: "620e95cf-d32f-4ce8-b614-14a27fee162e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:19.519214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.519182    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume\") pod \"e2cb7728-be28-4fbd-8695-1b97253b27ae\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") "
Nov 01 20:20:19.519418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.519235    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials\") pod \"e2cb7728-be28-4fbd-8695-1b97253b27ae\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") "
Nov 01 20:20:19.519496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.519426    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv\") pod \"e2cb7728-be28-4fbd-8695-1b97253b27ae\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") "
Nov 01 20:20:19.519616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.519588    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume" pod "e2cb7728-be28-4fbd-8695-1b97253b27ae" (UID: "e2cb7728-be28-4fbd-8695-1b97253b27ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:19.519782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.519600    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume\") pod \"e2cb7728-be28-4fbd-8695-1b97253b27ae\" (UID: \"e2cb7728-be28-4fbd-8695-1b97253b27ae\") "
Nov 01 20:20:19.520004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.519984    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume" pod "e2cb7728-be28-4fbd-8695-1b97253b27ae" (UID: "e2cb7728-be28-4fbd-8695-1b97253b27ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:19.520318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.520300    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.520386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.520322    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9b29\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-kube-api-access-v9b29\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.520386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.520337    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e95cf-d32f-4ce8-b614-14a27fee162e-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.520386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.520351    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cb7728-be28-4fbd-8695-1b97253b27ae-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.521462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.521442    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials" pod "e2cb7728-be28-4fbd-8695-1b97253b27ae" (UID: "e2cb7728-be28-4fbd-8695-1b97253b27ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:19.521520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.521466    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv" pod "e2cb7728-be28-4fbd-8695-1b97253b27ae" (UID: "e2cb7728-be28-4fbd-8695-1b97253b27ae"). InnerVolumeSpecName "kube-api-access-9wzfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:19.591140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.591101    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:19.621557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.621522    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzfv\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-kube-api-access-9wzfv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.621557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.621552    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2cb7728-be28-4fbd-8695-1b97253b27ae-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:19.648127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.648090    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"]
Nov 01 20:20:19.649158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649128    6479 cpu_manager.go:414] "containerMap: removing container" podUID="03ebc8bc-6a2e-42df-9e29-72e05fc5886b" containerName="maas-api"
Nov 01 20:20:19.649158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649157    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ebc8bc-6a2e-42df-9e29-72e05fc5886b" containerName="maas-api"
Nov 01 20:20:19.649399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649239    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e2cb7728-be28-4fbd-8695-1b97253b27ae" containerName="payload-processing"
Nov 01 20:20:19.649399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649249    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cb7728-be28-4fbd-8695-1b97253b27ae" containerName="payload-processing"
Nov 01 20:20:19.649399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649264    6479 cpu_manager.go:414] "containerMap: removing container" podUID="0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" containerName="payload-pre-processing"
Nov 01 20:20:19.649399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649274    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" containerName="payload-pre-processing"
Nov 01 20:20:19.649399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649314    6479 cpu_manager.go:414] "containerMap: removing container" podUID="620e95cf-d32f-4ce8-b614-14a27fee162e" containerName="payload-processing"
Nov 01 20:20:19.649399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649322    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620e95cf-d32f-4ce8-b614-14a27fee162e" containerName="payload-processing"
Nov 01 20:20:19.649730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649485    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cb7728-be28-4fbd-8695-1b97253b27ae" containerName="payload-processing"
Nov 01 20:20:19.649730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649499    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd13237-3bf0-4a92-a6dd-30e8f7ef0fcf" containerName="payload-pre-processing"
Nov 01 20:20:19.649730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649514    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="620e95cf-d32f-4ce8-b614-14a27fee162e" containerName="payload-processing"
Nov 01 20:20:19.649730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.649525    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ebc8bc-6a2e-42df-9e29-72e05fc5886b" containerName="maas-api"
Nov 01 20:20:19.655261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.655235    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.664880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.664796    6479 generic.go:356] "Generic (PLEG): container finished" podID="620e95cf-d32f-4ce8-b614-14a27fee162e" containerID="087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8" exitCode=0
Nov 01 20:20:19.664880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.664864    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"
Nov 01 20:20:19.665093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.664983    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz" event={"ID":"620e95cf-d32f-4ce8-b614-14a27fee162e","Type":"ContainerDied","Data":"087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8"}
Nov 01 20:20:19.665093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.665032    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz" event={"ID":"620e95cf-d32f-4ce8-b614-14a27fee162e","Type":"ContainerDied","Data":"7eaf172114d2e3d4f95af8495456947a327f4c9aa002d1d04ae95b6abccce35b"}
Nov 01 20:20:19.665093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.665063    6479 scope.go:122] "RemoveContainer" containerID="087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8"
Nov 01 20:20:19.667312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.667288    6479 generic.go:356] "Generic (PLEG): container finished" podID="e2cb7728-be28-4fbd-8695-1b97253b27ae" containerID="5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186" exitCode=0
Nov 01 20:20:19.667580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.667561    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs" event={"ID":"e2cb7728-be28-4fbd-8695-1b97253b27ae","Type":"ContainerDied","Data":"5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186"}
Nov 01 20:20:19.667868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.667849    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs" event={"ID":"e2cb7728-be28-4fbd-8695-1b97253b27ae","Type":"ContainerDied","Data":"f0acd43482837959bc88893322132faf9d6d0ba2581472b89e988d8db7373eda"}
Nov 01 20:20:19.668041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.668026    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"
Nov 01 20:20:19.682399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.682369    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"]
Nov 01 20:20:19.682563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.682543    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8b6cd6-openshift-default-dockercfg-tnb8t\""
Nov 01 20:20:19.690404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.688554    6479 scope.go:122] "RemoveContainer" containerID="087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8"
Nov 01 20:20:19.690404 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:19.689026    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8\": container with ID starting with 087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8 not found: ID does not exist" containerID="087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8"
Nov 01 20:20:19.690404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.689082    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8"} err="failed to get container status \"087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8\": rpc error: code = NotFound desc = could not find container \"087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8\": container with ID starting with 087ee2e5a2703dc4504818ac0ccfb4eec9645fe0b09cf0943e9e00b7fa38bcc8 not found: ID does not exist"
Nov 01 20:20:19.690404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.689111    6479 scope.go:122] "RemoveContainer" containerID="5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186"
Nov 01 20:20:19.709981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.709938    6479 scope.go:122] "RemoveContainer" containerID="5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186"
Nov 01 20:20:19.710558 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:19.710528    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186\": container with ID starting with 5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186 not found: ID does not exist" containerID="5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186"
Nov 01 20:20:19.710738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.710565    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186"} err="failed to get container status \"5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186\": rpc error: code = NotFound desc = could not find container \"5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186\": container with ID starting with 5040f27c0eea8f403f176e61d80942a982b71368d40ab24f31d05dc4d6370186 not found: ID does not exist"
Nov 01 20:20:19.722573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.722527    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.722725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.722590    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.722816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.722731    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.722878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.722823    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.722938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.722877    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.723038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.722994    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.723142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.723036    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2sg\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.723279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.723253    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.723396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.723292    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.728771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.728744    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"]
Nov 01 20:20:19.733822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.733798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:19.735921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.735901    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-845fb4dc4b-s2wjz"]
Nov 01 20:20:19.751330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.751304    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"]
Nov 01 20:20:19.757634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.757605    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7854fcbcdd-m6rzs"]
Nov 01 20:20:19.821252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.821215    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"]
Nov 01 20:20:19.824835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.824793    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.824853    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2sg\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825101    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825133    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825164    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825222    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825256    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825336    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.825405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.825393    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.828323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.828295    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.828685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.828626    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.828808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.828730    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.828808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.828301    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.828934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.828889    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.829338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.829313    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.829435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.829352    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.831064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.831044    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.838052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.838024    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a6873b-openshift-default-dockercfg-hfgdz\""
Nov 01 20:20:19.840785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.840760    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.841227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.841202    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2sg\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:19.843729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.843702    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"]
Nov 01 20:20:19.843973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.843944    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.927851    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.927925    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928038    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928114    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928146    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928222    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928252    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfnz\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928369    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.976724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.928418    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:19.977941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:19.976811    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:20.001063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.000784    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:20.060494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.036913    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.060798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.034634    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.061224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.061111    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.061224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.061186    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.061672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.061550    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.061829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.061637    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfnz\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.062220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.062154    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.062461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.062383    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.064809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.062744    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.064809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.062793    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.066770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.066744    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.068245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.068193    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.068637 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.068620    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.069233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.069211    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.120824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.094568    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.120824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.094804    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.120824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.120194    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.148530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.148451    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfnz\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.164409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.164342    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:20.229266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.229182    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:20.359320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.359223    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:20.512807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.512772    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"]
Nov 01 20:20:20.515425 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:20.515313    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6263de7_73f1_434c_bf75_230cc71de2f8.slice/crio-8fe11b5b2f3a4628961e34bcef55d7d515070c533a7f8e6cfe25a33e1881d960 WatchSource:0}: Error finding container 8fe11b5b2f3a4628961e34bcef55d7d515070c533a7f8e6cfe25a33e1881d960: Status 404 returned error can't find the container with id 8fe11b5b2f3a4628961e34bcef55d7d515070c533a7f8e6cfe25a33e1881d960
Nov 01 20:20:20.522768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.522730    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:20:20.522995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.522963    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:20:20.523094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.523029    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:20:20.527085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.527060    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"]
Nov 01 20:20:20.528196 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:20.528173    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c65fc48_3041_4a60_a9da_b7ed8ab4d83c.slice/crio-af70a6f4751606fec548c53dad42473d82e43c23f1febd97656cb9fd9e86983c WatchSource:0}: Error finding container af70a6f4751606fec548c53dad42473d82e43c23f1febd97656cb9fd9e86983c: Status 404 returned error can't find the container with id af70a6f4751606fec548c53dad42473d82e43c23f1febd97656cb9fd9e86983c
Nov 01 20:20:20.533628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.533599    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:20:20.533777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.533691    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:20:20.533777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.533722    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:20:20.727439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.727396    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620e95cf-d32f-4ce8-b614-14a27fee162e" path="/var/lib/kubelet/pods/620e95cf-d32f-4ce8-b614-14a27fee162e/volumes"
Nov 01 20:20:20.728563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.728519    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2cb7728-be28-4fbd-8695-1b97253b27ae" path="/var/lib/kubelet/pods/e2cb7728-be28-4fbd-8695-1b97253b27ae/volumes"
Nov 01 20:20:20.729311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.729270    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" event={"ID":"c6263de7-73f1-434c-bf75-230cc71de2f8","Type":"ContainerStarted","Data":"4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"}
Nov 01 20:20:20.729497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.729469    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" event={"ID":"c6263de7-73f1-434c-bf75-230cc71de2f8","Type":"ContainerStarted","Data":"8fe11b5b2f3a4628961e34bcef55d7d515070c533a7f8e6cfe25a33e1881d960"}
Nov 01 20:20:20.729629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.729613    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" event={"ID":"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c","Type":"ContainerStarted","Data":"e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"}
Nov 01 20:20:20.729790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.729776    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" event={"ID":"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c","Type":"ContainerStarted","Data":"af70a6f4751606fec548c53dad42473d82e43c23f1febd97656cb9fd9e86983c"}
Nov 01 20:20:20.735157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.735107    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podStartSLOduration=1.735090414 podStartE2EDuration="1.735090414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:20.730250414 +0000 UTC m=+1266.145882887" watchObservedRunningTime="2026-11-01 20:20:20.735090414 +0000 UTC m=+1266.150722890"
Nov 01 20:20:20.769367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.769246    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:20.774084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.774007    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podStartSLOduration=1.7739831640000001 podStartE2EDuration="1.773983164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:20.766267565 +0000 UTC m=+1266.181900040" watchObservedRunningTime="2026-11-01 20:20:20.773983164 +0000 UTC m=+1266.189615639"
Nov 01 20:20:20.841901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.841863    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"]
Nov 01 20:20:20.847977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.847944    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:20.854697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.854667    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-933245\""
Nov 01 20:20:20.877215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.877134    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:20.881342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.881292    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"]
Nov 01 20:20:20.899168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.899129    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"]
Nov 01 20:20:20.923288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:20.923239    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:21.068238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.068130    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-933245-dockercfg-5bprb\""
Nov 01 20:20:21.207789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.074552    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvzq\" (UniqueName: \"kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq\") pod \"payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:21.207789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.199830    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:20:21.207789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.199905    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:20:21.228750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.210131    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:20:21.228750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.210202    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:20:21.282443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.280393    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:21.282443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.280456    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"]
Nov 01 20:20:21.282443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.280511    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:21.282443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.280638    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.391710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.389433    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:21.460261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.460043    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvzq\" (UniqueName: \"kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq\") pod \"payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:21.460854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.460829    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:21.465998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.465963    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume\") pod \"payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") " pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.466115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.466048    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhk7\" (UniqueName: \"kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7\") pod \"payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") " pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.467153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.467085    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:21.470101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.470078    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"]
Nov 01 20:20:21.476070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.476051    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvzq\" (UniqueName: \"kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq\") pod \"payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:21.477224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.477206    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f347a907-ckw8z"]
Nov 01 20:20:21.567099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.567056    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume\") pod \"payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") " pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.567307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.567118    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhk7\" (UniqueName: \"kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7\") pod \"payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") " pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.567946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.567923    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume\") pod \"payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") " pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.575152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.575125    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhk7\" (UniqueName: \"kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7\") pod \"payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") " pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.770235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.770179    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:21.771310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:21.771283    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:22.010210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.010177    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:22.069711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.068509    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"]
Nov 01 20:20:22.072535 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:22.072505    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8880dea5_99d5_4512_ab04_744f461b1aec.slice/crio-8c3b6b1d44851c852f3e63ffc001ef7bf9b48fcc71b6a9a24dd15c956222b7b1 WatchSource:0}: Error finding container 8c3b6b1d44851c852f3e63ffc001ef7bf9b48fcc71b6a9a24dd15c956222b7b1: Status 404 returned error can't find the container with id 8c3b6b1d44851c852f3e63ffc001ef7bf9b48fcc71b6a9a24dd15c956222b7b1
Nov 01 20:20:22.108715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.108681    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"]
Nov 01 20:20:22.110810 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:22.110782    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18d37c2_8f32_4933_84d9_691b54067543.slice/crio-3f3f89853c63ed025a0863c53f14c2218e44ed8b90d4f07a7c697b94bba540bb WatchSource:0}: Error finding container 3f3f89853c63ed025a0863c53f14c2218e44ed8b90d4f07a7c697b94bba540bb: Status 404 returned error can't find the container with id 3f3f89853c63ed025a0863c53f14c2218e44ed8b90d4f07a7c697b94bba540bb
Nov 01 20:20:22.168735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.168632    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:22.176124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.172968    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:22.457390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.457352    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:22.709410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.709328    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9" path="/var/lib/kubelet/pods/a3b467fc-cf1a-4b26-8b22-2ce5041a2ff9/volumes"
Nov 01 20:20:22.723730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.723690    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts" event={"ID":"8880dea5-99d5-4512-ab04-744f461b1aec","Type":"ContainerStarted","Data":"1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368"}
Nov 01 20:20:22.723928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.723750    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts" event={"ID":"8880dea5-99d5-4512-ab04-744f461b1aec","Type":"ContainerStarted","Data":"8c3b6b1d44851c852f3e63ffc001ef7bf9b48fcc71b6a9a24dd15c956222b7b1"}
Nov 01 20:20:22.723928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.723806    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:22.725524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.725494    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq" event={"ID":"e18d37c2-8f32-4933-84d9-691b54067543","Type":"ContainerStarted","Data":"713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d"}
Nov 01 20:20:22.725675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.725549    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:22.725675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.725573    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq" event={"ID":"e18d37c2-8f32-4933-84d9-691b54067543","Type":"ContainerStarted","Data":"3f3f89853c63ed025a0863c53f14c2218e44ed8b90d4f07a7c697b94bba540bb"}
Nov 01 20:20:22.725968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.725943    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:22.726077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.725997    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:22.727275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.727255    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:20:22.727364 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.727311    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:20:22.742085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.742023    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts" podStartSLOduration=2.742006374 podStartE2EDuration="2.742006374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:22.740312994 +0000 UTC m=+1268.155945467" watchObservedRunningTime="2026-11-01 20:20:22.742006374 +0000 UTC m=+1268.157638849"
Nov 01 20:20:22.799812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.799090    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq" podStartSLOduration=2.799068594 podStartE2EDuration="2.799068594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:22.797639359 +0000 UTC m=+1268.213271835" watchObservedRunningTime="2026-11-01 20:20:22.799068594 +0000 UTC m=+1268.214701068"
Nov 01 20:20:22.907894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:22.907851    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:23.121723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:23.121074    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:23.305360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:23.305323    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:23.410618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:23.410531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:23.511067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:23.511035    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:23.617015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:23.616974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:24.202898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.202853    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:24.302505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.302466    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:24.304679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.304623    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"]
Nov 01 20:20:24.322906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.322869    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"]
Nov 01 20:20:24.323082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.322995    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.333743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.333553    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a6873b-dockercfg-sjfqm\""
Nov 01 20:20:24.333743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.333691    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a6873b\""
Nov 01 20:20:24.428671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.425229    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.428671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.425288    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg59p\" (UniqueName: \"kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p\") pod \"payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.531718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.529587    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg59p\" (UniqueName: \"kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p\") pod \"payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.531718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.529984    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.531718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.530839    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.590718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.589591    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg59p\" (UniqueName: \"kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p\") pod \"payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.652670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.644072    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:24.848892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.848807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:24.992761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:24.992708    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"]
Nov 01 20:20:24.996436 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:24.996399    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f79e45c_d324_43ef_a7cd_efedb4fd9a8d.slice/crio-3d790533fd8a178ea1243dce327b903d6bf415a8c1bb816c8ad977da95dbf64d WatchSource:0}: Error finding container 3d790533fd8a178ea1243dce327b903d6bf415a8c1bb816c8ad977da95dbf64d: Status 404 returned error can't find the container with id 3d790533fd8a178ea1243dce327b903d6bf415a8c1bb816c8ad977da95dbf64d
Nov 01 20:20:25.180962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.180924    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:25.222106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.222062    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"]
Nov 01 20:20:25.227093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.227068    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.235451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.235416    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"]
Nov 01 20:20:25.237054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.237026    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8b6cd6\""
Nov 01 20:20:25.239277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.239254    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"]
Nov 01 20:20:25.244417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.244394    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.253602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.253568    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"]
Nov 01 20:20:25.255755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.255714    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qpj\" (UniqueName: \"kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj\") pod \"payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.255992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.255972    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.270128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.270100    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8b6cd6-dockercfg-kl2m2\""
Nov 01 20:20:25.273728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.273695    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:25.362694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.362576    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.363105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.363076    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qpj\" (UniqueName: \"kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj\") pod \"payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.372670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.363279    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.373065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.373036    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7jd\" (UniqueName: \"kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd\") pod \"payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.375278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.374932    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.421670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.416895    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qpj\" (UniqueName: \"kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj\") pod \"payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.474593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.474546    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jd\" (UniqueName: \"kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd\") pod \"payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.478939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.478910    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.480264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.480237    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.507717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.507444    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7jd\" (UniqueName: \"kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd\") pod \"payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.551034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.550989    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:25.582234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.582182    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:25.753314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.753268    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w" event={"ID":"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d","Type":"ContainerStarted","Data":"954f7ee743b6156d9018b21fc81ca91a1fc58f1b094d2235a67f749ebf9d3ab7"}
Nov 01 20:20:25.753493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.753341    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w" event={"ID":"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d","Type":"ContainerStarted","Data":"3d790533fd8a178ea1243dce327b903d6bf415a8c1bb816c8ad977da95dbf64d"}
Nov 01 20:20:25.754513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.754441    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:25.786883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.786812    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w" podStartSLOduration=1.786791279 podStartE2EDuration="1.786791279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:25.781731959 +0000 UTC m=+1271.197364433" watchObservedRunningTime="2026-11-01 20:20:25.786791279 +0000 UTC m=+1271.202423753"
Nov 01 20:20:25.841067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.841034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:25.911422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.911390    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"]
Nov 01 20:20:25.912074 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:25.912027    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2f7d92_05fc_4a3d_95ba_beea9241bab6.slice/crio-fc40b6550528eb068e096874c4ff6b46053cb5e616eb845a4cd6f3b36117f080 WatchSource:0}: Error finding container fc40b6550528eb068e096874c4ff6b46053cb5e616eb845a4cd6f3b36117f080: Status 404 returned error can't find the container with id fc40b6550528eb068e096874c4ff6b46053cb5e616eb845a4cd6f3b36117f080
Nov 01 20:20:25.915550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:25.915521    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"]
Nov 01 20:20:25.916259 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:25.916235    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cce8947_6a76_450d_b0d9_38d96efe2a0c.slice/crio-05a4b78a892c94f9e7203da8a2d3a51e8b6ccd89ce912fdafb3583caeffb2c53 WatchSource:0}: Error finding container 05a4b78a892c94f9e7203da8a2d3a51e8b6ccd89ce912fdafb3583caeffb2c53: Status 404 returned error can't find the container with id 05a4b78a892c94f9e7203da8a2d3a51e8b6ccd89ce912fdafb3583caeffb2c53
Nov 01 20:20:26.279846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.279803    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:20:26.648393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.648305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:26.767215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.767169    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" event={"ID":"1cce8947-6a76-450d-b0d9-38d96efe2a0c","Type":"ContainerStarted","Data":"5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570"}
Nov 01 20:20:26.767396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.767233    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" event={"ID":"1cce8947-6a76-450d-b0d9-38d96efe2a0c","Type":"ContainerStarted","Data":"05a4b78a892c94f9e7203da8a2d3a51e8b6ccd89ce912fdafb3583caeffb2c53"}
Nov 01 20:20:26.767396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.767358    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:26.769165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.769137    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" event={"ID":"cc2f7d92-05fc-4a3d-95ba-beea9241bab6","Type":"ContainerStarted","Data":"0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32"}
Nov 01 20:20:26.769281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.769189    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" event={"ID":"cc2f7d92-05fc-4a3d-95ba-beea9241bab6","Type":"ContainerStarted","Data":"fc40b6550528eb068e096874c4ff6b46053cb5e616eb845a4cd6f3b36117f080"}
Nov 01 20:20:26.769347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.769296    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:26.794957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.794902    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" podStartSLOduration=1.794886963 podStartE2EDuration="1.794886963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:26.791308675 +0000 UTC m=+1272.206941149" watchObservedRunningTime="2026-11-01 20:20:26.794886963 +0000 UTC m=+1272.210519436"
Nov 01 20:20:26.840507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:26.840451    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" podStartSLOduration=1.840436889 podStartE2EDuration="1.840436889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:26.836893035 +0000 UTC m=+1272.252525507" watchObservedRunningTime="2026-11-01 20:20:26.840436889 +0000 UTC m=+1272.256069362"
Nov 01 20:20:28.043902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:28.043851    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:28.628107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:28.628068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:29.311671 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:29.311417    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:20:30.044251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:30.044215    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:31.248679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:31.248618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:31.942038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:31.941997    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:32.562634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:32.562601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:33.326704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:33.326608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:33.405284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:33.405239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:33.556291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:33.556235    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:33.735848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:33.735811    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:20:33.736249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:33.735883    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:20:34.197895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:34.197857    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:34.955236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:34.955198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:35.565737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:35.565697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:36.196664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.196614    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"]
Nov 01 20:20:36.201553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.201529    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.223731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.207836    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a6873b-serving-cert\""
Nov 01 20:20:36.223731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.222060    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"]
Nov 01 20:20:36.243542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.243508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:36.294176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.294137    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rr4\" (UniqueName: \"kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.294348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.294199    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.294389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.294357    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.294423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.294387    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.349713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.347331    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:36.400672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.397023    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rr4\" (UniqueName: \"kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.400672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.397089    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.400672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.397227    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.400672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.397253    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.401635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.401599    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.402019 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:36.401965    6479 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-serving-cert: secret "maas-api-e2e-worker-w1-a6873b-serving-cert" not found
Nov 01 20:20:36.402335 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:36.402322    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls podName:cf650dd7-0480-4237-9ac5-2267acad84d5 nodeName:}" failed. No retries permitted until 2026-11-01 20:20:36.902209276 +0000 UTC m=+1282.317841733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls") pod "maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" (UID: "cf650dd7-0480-4237-9ac5-2267acad84d5") : secret "maas-api-e2e-worker-w1-a6873b-serving-cert" not found
Nov 01 20:20:36.411381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.411347    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.419166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.419134    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rr4\" (UniqueName: \"kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.508541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.508506    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:36.903809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.903722    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:36.906975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:36.906479    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:37.147751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.147712    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:37.188315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.188222    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:37.327324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.325046    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:37.394882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.394842    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:37.453433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.453391    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"]
Nov 01 20:20:37.454357 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:20:37.454318    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf650dd7_0480_4237_9ac5_2267acad84d5.slice/crio-7b211625b11094dac305d54a28e8184d0eedec7c8fdbf7be0c95300e05bf02e5 WatchSource:0}: Error finding container 7b211625b11094dac305d54a28e8184d0eedec7c8fdbf7be0c95300e05bf02e5: Status 404 returned error can't find the container with id 7b211625b11094dac305d54a28e8184d0eedec7c8fdbf7be0c95300e05bf02e5
Nov 01 20:20:37.831879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.831771    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:20:37.837560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.833059    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:20:37.837560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.834535    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:20:37.939149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:37.939096    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" event={"ID":"cf650dd7-0480-4237-9ac5-2267acad84d5","Type":"ContainerStarted","Data":"7b211625b11094dac305d54a28e8184d0eedec7c8fdbf7be0c95300e05bf02e5"}
Nov 01 20:20:38.141862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:38.141773    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:38.739794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:38.739755    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:38.948684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:38.948619    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" event={"ID":"cf650dd7-0480-4237-9ac5-2267acad84d5","Type":"ContainerStarted","Data":"0f9ec6c3f65dcde12b5d81bfaab206b14e8b937e596a9db76a252a8aba3186fd"}
Nov 01 20:20:38.948843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:38.948717    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:38.967490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:38.967425    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" podStartSLOduration=2.461500382 podStartE2EDuration="2.967410328s" totalImagesPullingTime="505.909946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:38.96368446 +0000 UTC m=+1284.379316930" watchObservedRunningTime="2026-11-01 20:20:38.967410328 +0000 UTC m=+1284.383042801"
Nov 01 20:20:39.438624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:39.438587    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:40.168802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:40.168755    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:40.829879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:40.829832    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:41.447462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:41.447414    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:42.141552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:42.141506    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:42.836985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:42.836950    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:43.429714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:43.429677    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:44.173824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:44.173779    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:44.766468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:44.766423    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:44.973814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:44.973782    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:20:45.688252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:45.688207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:46.517778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:46.517737    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:47.240850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:47.240810    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:47.534359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:47.534261    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:20:48.066005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:48.065966    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:48.655023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:48.654894    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:48.773694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:48.773638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:49.718279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:49.718241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:49.892991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:49.892951    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:50.981670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:50.976594    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:51.324347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:51.324189    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:51.986596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:51.986551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:52.821209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:52.821176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:53.218683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:53.218350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:53.427664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:53.427614    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:53.704927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:53.702725    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:53.800308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:53.800269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:53.901363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:53.901324    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:53.996713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:53.996612    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:54.645523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:54.645490    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:54.866057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:54.866019    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:54.973926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:54.973877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:55.531005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:55.530924    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:56.359055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:56.359016    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:57.225472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:57.225430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:57.849140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:57.849093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:57.995273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:57.995188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:58.261292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:58.261243    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:58.953396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:58.953307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:59.190003 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:20:59.189935    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:20:59.237876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.237785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:59.252974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.252938    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"]
Nov 01 20:20:59.260206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.259198    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.267868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.266972    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-036c69\""
Nov 01 20:20:59.270606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.270577    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"]
Nov 01 20:20:59.330620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.330584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:20:59.498826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.466614    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g89p\" (UniqueName: \"kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p\") pod \"payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.502303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.501997    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.631725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.613313    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g89p\" (UniqueName: \"kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p\") pod \"payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.631725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.613379    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.631725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.614732    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.685669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.684362    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g89p\" (UniqueName: \"kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p\") pod \"payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.914252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.914165    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:20:59.952177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:20:59.952136    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:00.054099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:00.053934    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:00.426475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:00.406459    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"]
Nov 01 20:21:00.445918 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:21:00.445523    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f68aa6_d7bc_4230_b5ea_6bcf505a7f53.slice/crio-863e9c3d1574710ff0a48d0816792199be52ad7443ea0363c7c128d422c732b1 WatchSource:0}: Error finding container 863e9c3d1574710ff0a48d0816792199be52ad7443ea0363c7c128d422c732b1: Status 404 returned error can't find the container with id 863e9c3d1574710ff0a48d0816792199be52ad7443ea0363c7c128d422c732b1
Nov 01 20:21:00.561686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:00.561559    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" event={"ID":"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53","Type":"ContainerStarted","Data":"863e9c3d1574710ff0a48d0816792199be52ad7443ea0363c7c128d422c732b1"}
Nov 01 20:21:00.867450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:00.867406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:01.573182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.573149    6479 generic.go:356] "Generic (PLEG): container finished" podID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerID="4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5" exitCode=0
Nov 01 20:21:01.573691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.573236    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" event={"ID":"c6263de7-73f1-434c-bf75-230cc71de2f8","Type":"ContainerDied","Data":"4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"}
Nov 01 20:21:01.573869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.573846    6479 scope.go:122] "RemoveContainer" containerID="4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"
Nov 01 20:21:01.575470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.575402    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" event={"ID":"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53","Type":"ContainerStarted","Data":"b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772"}
Nov 01 20:21:01.575470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.575470    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:21:01.575973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.575944    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:21:01.576074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.576028    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:21:01.576138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.576076    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:21:01.578186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.578160    6479 generic.go:356] "Generic (PLEG): container finished" podID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerID="e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06" exitCode=0
Nov 01 20:21:01.578291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.578265    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" event={"ID":"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c","Type":"ContainerDied","Data":"e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"}
Nov 01 20:21:01.578631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.578615    6479 scope.go:122] "RemoveContainer" containerID="e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"
Nov 01 20:21:01.580182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.580153    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:21:01.580264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.580233    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:21:01.580344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.580282    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:21:01.599008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.598954    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:01.643398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.643329    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" podStartSLOduration=2.643308756 podStartE2EDuration="2.643308756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:01.639813356 +0000 UTC m=+1307.055445829" watchObservedRunningTime="2026-11-01 20:21:01.643308756 +0000 UTC m=+1307.058941229"
Nov 01 20:21:01.681695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.678875    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"]
Nov 01 20:21:01.711250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:01.711217    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:02.611362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:02.611320    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" event={"ID":"c6263de7-73f1-434c-bf75-230cc71de2f8","Type":"ContainerStarted","Data":"1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829"}
Nov 01 20:21:02.617863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:02.617805    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" event={"ID":"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c","Type":"ContainerStarted","Data":"e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903"}
Nov 01 20:21:02.692667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:02.692412    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:03.023636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.021976    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:21:03.025013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.024985    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:21:03.025169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.025037    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:21:03.173671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.167049    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:21:03.173671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.173044    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:21:03.173671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.173100    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:21:03.597403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:03.597363    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:04.009708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:04.008086    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:21:04.009708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:04.008152    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:21:04.181153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:04.181110    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:21:04.181444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:04.181410    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:21:04.284728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:04.284518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:04.967630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:04.967579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:05.007321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:05.007127    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:21:05.007321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:05.007204    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:21:05.168672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:05.168074    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:21:05.168672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:05.168136    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:21:05.690086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:05.690046    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:06.008662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:06.008019    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:21:06.008662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:06.008098    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:21:06.166088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:06.166049    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:21:06.166364 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:06.166334    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:21:06.373690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:06.373530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:06.689730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:06.688722    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk" podUID="8193030a-964c-4f08-9d82-9feeb0085864" containerName="payload-processing" containerID="cri-o://456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2" gracePeriod=25
Nov 01 20:21:07.084087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.072809    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:21:07.084087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.072870    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:21:07.192672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.189949    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:07.318984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.245010    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:21:07.318984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.245062    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:21:07.419480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.416904    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:07.531671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.531524    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:07.613584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.613511    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:21:07.713695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.710840    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume\") pod \"8193030a-964c-4f08-9d82-9feeb0085864\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") "
Nov 01 20:21:07.713695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.710908    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials\") pod \"8193030a-964c-4f08-9d82-9feeb0085864\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") "
Nov 01 20:21:07.713695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.711115    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume\") pod \"8193030a-964c-4f08-9d82-9feeb0085864\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") "
Nov 01 20:21:07.713695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.711203    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj\") pod \"8193030a-964c-4f08-9d82-9feeb0085864\" (UID: \"8193030a-964c-4f08-9d82-9feeb0085864\") "
Nov 01 20:21:07.713695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.712877    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume" pod "8193030a-964c-4f08-9d82-9feeb0085864" (UID: "8193030a-964c-4f08-9d82-9feeb0085864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:07.713695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.713295    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume" pod "8193030a-964c-4f08-9d82-9feeb0085864" (UID: "8193030a-964c-4f08-9d82-9feeb0085864"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:07.716788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.716136    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials" pod "8193030a-964c-4f08-9d82-9feeb0085864" (UID: "8193030a-964c-4f08-9d82-9feeb0085864"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:07.717352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.717325    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj" pod "8193030a-964c-4f08-9d82-9feeb0085864" (UID: "8193030a-964c-4f08-9d82-9feeb0085864"). InnerVolumeSpecName "kube-api-access-d5zvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:07.726685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.726436    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:21:07.726801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.726767    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:21:07.745014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.744991    6479 generic.go:356] "Generic (PLEG): container finished" podID="8193030a-964c-4f08-9d82-9feeb0085864" containerID="456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2" exitCode=0
Nov 01 20:21:07.745129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.745032    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"
Nov 01 20:21:07.745129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.745066    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk" event={"ID":"8193030a-964c-4f08-9d82-9feeb0085864","Type":"ContainerDied","Data":"456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2"}
Nov 01 20:21:07.745129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.745117    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk" event={"ID":"8193030a-964c-4f08-9d82-9feeb0085864","Type":"ContainerDied","Data":"9ebe338704f365c1c76c133e63d0ca38375beffdb9b7e3459c6583c5462cbceb"}
Nov 01 20:21:07.745235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.745141    6479 scope.go:122] "RemoveContainer" containerID="456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2"
Nov 01 20:21:07.827757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.827723    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"]
Nov 01 20:21:07.828145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.828118    6479 scope.go:122] "RemoveContainer" containerID="456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2"
Nov 01 20:21:07.828466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.828426    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:07.829001 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:07.828967    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2\": container with ID starting with 456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2 not found: ID does not exist" containerID="456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2"
Nov 01 20:21:07.829087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.829011    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2"} err="failed to get container status \"456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2\": rpc error: code = NotFound desc = could not find container \"456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2\": container with ID starting with 456425af140e273ff2fc37ea28a410ae011ac11e79361dcff19a50349beee3f2 not found: ID does not exist"
Nov 01 20:21:07.830023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.829998    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:07.830131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.830027    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5zvj\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-kube-api-access-d5zvj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:07.830131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.830044    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8193030a-964c-4f08-9d82-9feeb0085864-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:07.830131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.830058    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8193030a-964c-4f08-9d82-9feeb0085864-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:07.854671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:07.834793    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5b8dt8jfk"]
Nov 01 20:21:08.014667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:08.013908    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 01 20:21:08.014667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:08.013977    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 01 20:21:08.190672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:08.189781    6479 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 20:21:08.190672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:08.189863    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 20:21:08.405285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:08.405181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:08.709583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:08.709548    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8193030a-964c-4f08-9d82-9feeb0085864" path="/var/lib/kubelet/pods/8193030a-964c-4f08-9d82-9feeb0085864/volumes"
Nov 01 20:21:09.011062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.010989    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:21:09.012190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.012170    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:21:09.058614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.058554    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:09.238782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.238109    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:21:09.246088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.241759    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:21:09.796444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.796409    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"]
Nov 01 20:21:09.797413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.797388    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8193030a-964c-4f08-9d82-9feeb0085864" containerName="payload-processing"
Nov 01 20:21:09.797413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.797412    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8193030a-964c-4f08-9d82-9feeb0085864" containerName="payload-processing"
Nov 01 20:21:09.797658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.797630    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8193030a-964c-4f08-9d82-9feeb0085864" containerName="payload-processing"
Nov 01 20:21:09.803014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.802988    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:09.822577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.822548    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-036c69-serving-cert\""
Nov 01 20:21:09.826122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.826094    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"]
Nov 01 20:21:09.878868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.878822    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:09.904676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.904092    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:09.904676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.904240    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdfv\" (UniqueName: \"kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:09.904676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.904267    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:09.904676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.904306    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.002673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:09.997524    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:10.009666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.006679    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdfv\" (UniqueName: \"kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.009666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.006731    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.009666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.006778    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.009666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.007093    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.009666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.008372    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.041728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.040398    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdfv\" (UniqueName: \"kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.041728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.040964    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.048371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.048296    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.090966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.090933    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:10.118268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.118242    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:10.476957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.458860    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"]
Nov 01 20:21:10.765629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.765513    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:10.773465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:10.773435    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j" event={"ID":"462fa0ea-b9e2-4515-89b6-bc93de691a95","Type":"ContainerStarted","Data":"788944ed943d2216e92c2e7ef986f23d93206e7079965a7c80426fa7b8baec9c"}
Nov 01 20:21:11.400804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:11.400713    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:11.506134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:11.496064    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:11.807822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:11.805182    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j" event={"ID":"462fa0ea-b9e2-4515-89b6-bc93de691a95","Type":"ContainerStarted","Data":"15a24c6349570e44fe063516a89921df66da80bf3dcf5307016be7d9d382006b"}
Nov 01 20:21:11.807822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:11.806258    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:11.884671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:11.877125    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j" podStartSLOduration=2.357250011 podStartE2EDuration="2.877102761s" totalImagesPullingTime="519.85275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:11.834265823 +0000 UTC m=+1317.249898296" watchObservedRunningTime="2026-11-01 20:21:11.877102761 +0000 UTC m=+1317.292735235"
Nov 01 20:21:12.207669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:12.206987    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:12.214791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:12.212877    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"]
Nov 01 20:21:12.214791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:12.213182    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy" containerID="cri-o://302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74" gracePeriod=30
Nov 01 20:21:12.434573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:12.434539    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:12.551729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:12.545785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:12.657479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:12.652953    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:21:13.366813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:13.366777    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:15.756144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:15.756106    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:15.856548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:15.856503    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:16.058826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.058731    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:21:16.259232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.259195    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:16.504446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.504412    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"]
Nov 01 20:21:16.510621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.510595    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.525250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.525181    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"default-dockercfg-vklkk\""
Nov 01 20:21:16.527572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.526629    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"premium-w1-a6873b-kserve-self-signed-certs\""
Nov 01 20:21:16.527806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.527786    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"kube-root-ca.crt\""
Nov 01 20:21:16.527887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.527839    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"openshift-service-ca.crt\""
Nov 01 20:21:16.529821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.529793    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"]
Nov 01 20:21:16.622375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.622336    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.622375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.622379    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.622616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.622398    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvqq\" (UniqueName: \"kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.622616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.622522    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.622739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.622635    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.622780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.622738    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.698674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.689692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:16.729765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.729722    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.730512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.730488    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.731051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.731032    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvqq\" (UniqueName: \"kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.741670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.731200    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.742061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.742029    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.742353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.742333    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.743082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.730991    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.743232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.742500    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.743354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.742812    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.754108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.754074    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvqq\" (UniqueName: \"kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.773197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.773119    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.783668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.774733    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs\") pod \"premium-w1-a6873b-kserve-95dff6f9b-sxs96\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") " pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:16.842155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:16.831940    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:17.499037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.499000    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"]
Nov 01 20:21:17.638722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.638580    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:17.817779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.817741    6479 scope.go:122] "RemoveContainer" containerID="1fe7a3028b7352beb6e88a07af923bff24217361666606734c5c270e99359244"
Nov 01 20:21:17.922149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.922122    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:21:17.975539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.975497    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.975753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.975604    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.975821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.975793    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.975876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.975833    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.976077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.975930    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.976571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.976526    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:17.976812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.976785    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:17.976968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.976945    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.977147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.976950    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:17.977377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.977358    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.977872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.977687    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:17.978119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.978101    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.978662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.978629    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw\") pod \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\" (UID: \"093e6b04-5681-4f90-b80e-f7b0fbc7ce17\") "
Nov 01 20:21:17.979213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.978574    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:17.980077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.980054    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:17.980208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.980194    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:17.980301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.980289    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:17.980388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.980377    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:17.980468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.980457    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:17.981632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.981606    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:21:17.981786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.981764    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:17.983513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.983277    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "kube-api-access-jnwvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:17.983621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:17.983555    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token" pod "093e6b04-5681-4f90-b80e-f7b0fbc7ce17" (UID: "093e6b04-5681-4f90-b80e-f7b0fbc7ce17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:18.041344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.041263    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" event={"ID":"10f7f2ba-8bf9-470e-b738-000dcb16f3c4","Type":"ContainerStarted","Data":"01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323"}
Nov 01 20:21:18.041344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.041310    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" event={"ID":"10f7f2ba-8bf9-470e-b738-000dcb16f3c4","Type":"ContainerStarted","Data":"cf7a7fc1edc64851a1e992f46e5161df6ad2e0ba2f87c5da348f3644d260b127"}
Nov 01 20:21:18.042895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.042869    6479 generic.go:356] "Generic (PLEG): container finished" podID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerID="302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74" exitCode=0
Nov 01 20:21:18.043053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.042912    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" event={"ID":"093e6b04-5681-4f90-b80e-f7b0fbc7ce17","Type":"ContainerDied","Data":"302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74"}
Nov 01 20:21:18.043053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.042920    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"
Nov 01 20:21:18.043053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.042959    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7" event={"ID":"093e6b04-5681-4f90-b80e-f7b0fbc7ce17","Type":"ContainerDied","Data":"c0b57ba8beb0ff443b8dffaed3ed9258bdfa1a60f87202be6c19854927783fce"}
Nov 01 20:21:18.043053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.042990    6479 scope.go:122] "RemoveContainer" containerID="302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74"
Nov 01 20:21:18.068463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.068401    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" podStartSLOduration=2.06838186 podStartE2EDuration="2.06838186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:18.064395905 +0000 UTC m=+1323.480028379" watchObservedRunningTime="2026-11-01 20:21:18.06838186 +0000 UTC m=+1323.484014335"
Nov 01 20:21:18.083459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.083409    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwvw\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-kube-api-access-jnwvw\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:18.083459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.083454    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:18.083675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.083469    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:18.083675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.083483    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093e6b04-5681-4f90-b80e-f7b0fbc7ce17-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:18.086717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.086687    6479 scope.go:122] "RemoveContainer" containerID="302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74"
Nov 01 20:21:18.087071 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:18.087049    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74\": container with ID starting with 302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74 not found: ID does not exist" containerID="302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74"
Nov 01 20:21:18.087161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.087078    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74"} err="failed to get container status \"302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74\": rpc error: code = NotFound desc = could not find container \"302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74\": container with ID starting with 302f887e68331b17f33f568427a815b0c53a40e97180e4eef389e1a819c08a74 not found: ID does not exist"
Nov 01 20:21:18.095899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.095872    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"]
Nov 01 20:21:18.104837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.104812    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-mp2v7"]
Nov 01 20:21:18.365795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.365698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:18.709767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.709735    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" path="/var/lib/kubelet/pods/093e6b04-5681-4f90-b80e-f7b0fbc7ce17/volumes"
Nov 01 20:21:18.833194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.833161    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:18.926374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:18.926341    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:21:19.076365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:19.076281    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:19.958772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:19.958738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:20.638835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:20.638803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:21.231308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:21.231266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:21.858342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:21.858306    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:22.497815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:22.497770    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:22.598335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:22.598301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:23.346342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:23.346307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:24.318563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:24.318528    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:24.855367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:24.855330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:24.980673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:24.976783    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:25.317004 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:25.316975    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:26.236383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:26.236352    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:26.833090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:26.833047    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:26.847290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:26.847260    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:26.874026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:26.873988    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:27.151662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:27.151552    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:21:27.554982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:27.554941    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:28.102887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:28.102853    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:28.270884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:28.270848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:28.634829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:28.634787    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:28.729810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:28.729772    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:29.047307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.047267    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:29.136376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.126090    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-831cf1"]
Nov 01 20:21:29.138325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.138280    6479 cpu_manager.go:414] "containerMap: removing container" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy"
Nov 01 20:21:29.138547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.138523    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy"
Nov 01 20:21:29.138984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.138957    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="093e6b04-5681-4f90-b80e-f7b0fbc7ce17" containerName="istio-proxy"
Nov 01 20:21:29.171020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.170968    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-831cf1"]
Nov 01 20:21:29.171371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.171352    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:29.180950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.180914    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\""
Nov 01 20:21:29.240173 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:29.240133    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:21:29.325251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.325167    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66q4l\" (UniqueName: \"kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l\") pod \"test-curl-7426-831cf1\" (UID: \"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6\") " pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:29.426590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.426557    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66q4l\" (UniqueName: \"kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l\") pod \"test-curl-7426-831cf1\" (UID: \"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6\") " pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:29.434531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.434500    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66q4l\" (UniqueName: \"kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l\") pod \"test-curl-7426-831cf1\" (UID: \"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6\") " pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:29.515532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.515490    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:29.743712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.743682    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:29.794187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:29.794153    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-831cf1"]
Nov 01 20:21:29.795367 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:21:29.795328    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae9bbc4_dd3b_4476_b38f_9ecfaefe96b6.slice/crio-d0f50e343dcc9d5703181d8a4d282577aec49f902c7c1b822b35d5007dbe6892 WatchSource:0}: Error finding container d0f50e343dcc9d5703181d8a4d282577aec49f902c7c1b822b35d5007dbe6892: Status 404 returned error can't find the container with id d0f50e343dcc9d5703181d8a4d282577aec49f902c7c1b822b35d5007dbe6892
Nov 01 20:21:30.191081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.190994    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-831cf1" event={"ID":"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6","Type":"ContainerStarted","Data":"ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f"}
Nov 01 20:21:30.191081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.191043    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-831cf1" event={"ID":"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6","Type":"ContainerStarted","Data":"d0f50e343dcc9d5703181d8a4d282577aec49f902c7c1b822b35d5007dbe6892"}
Nov 01 20:21:30.207030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.206983    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7426-831cf1" podStartSLOduration=1.2069705530000001 podStartE2EDuration="1.206970553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:30.204530656 +0000 UTC m=+1335.620163131" watchObservedRunningTime="2026-11-01 20:21:30.206970553 +0000 UTC m=+1335.622603026"
Nov 01 20:21:30.347925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.347879    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:30.908150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.908113    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-831cf1"]
Nov 01 20:21:30.912610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.912580    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-831cf1"]
Nov 01 20:21:30.960755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:30.960692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:31.675053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:31.675000    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:32.206142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:32.206102    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7426-831cf1" podUID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" containerName="test-curl-7426-831cf1" containerID="cri-o://ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f" gracePeriod=2
Nov 01 20:21:32.208728 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:32.208693    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-831cf1\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:32.447408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:32.447373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:33.139232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:33.139200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:33.766597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:33.766563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:34.434341 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:34.434319    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:34.443173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:34.443144    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:34.560688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:34.559497    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l\" (UniqueName: \"kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l\") pod \"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6\" (UID: \"5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6\") "
Nov 01 20:21:34.563680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:34.563614    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l" pod "5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" (UID: "5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6"). InnerVolumeSpecName "kube-api-access-66q4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:34.661574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:34.661533    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66q4l\" (UniqueName: \"kubernetes.io/projected/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6-kube-api-access-66q4l\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:34.709250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:34.709217    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" path="/var/lib/kubelet/pods/5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6/volumes"
Nov 01 20:21:35.053592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.053556    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:35.255921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.255883    6479 generic.go:356] "Generic (PLEG): container finished" podID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" containerID="ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f" exitCode=137
Nov 01 20:21:35.255921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.255936    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-831cf1"
Nov 01 20:21:35.256181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.255938    6479 scope.go:122] "RemoveContainer" containerID="ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f"
Nov 01 20:21:35.266689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.266624    6479 scope.go:122] "RemoveContainer" containerID="ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f"
Nov 01 20:21:35.266978 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:35.266954    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f\": container with ID starting with ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f not found: ID does not exist" containerID="ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f"
Nov 01 20:21:35.267036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.266981    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f"} err="failed to get container status \"ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f\": rpc error: code = NotFound desc = could not find container \"ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f\": container with ID starting with ea62af2047b0d3582bcd39e4c0f6ff7e0d0d2a97995870b033b5f4e66595a84f not found: ID does not exist"
Nov 01 20:21:35.968570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:35.968534    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:36.911414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:36.905500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:37.004248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.004209    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:37.104272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.104214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:37.199980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.199943    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:37.320188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.320148    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:37.810251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.810217    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:37.911484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.911450    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"]
Nov 01 20:21:37.913380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.913352    6479 cpu_manager.go:414] "containerMap: removing container" podUID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" containerName="test-curl-7426-831cf1"
Nov 01 20:21:37.913517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.913381    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" containerName="test-curl-7426-831cf1"
Nov 01 20:21:37.913611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.913566    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae9bbc4-dd3b-4476-b38f-9ecfaefe96b6" containerName="test-curl-7426-831cf1"
Nov 01 20:21:37.919329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.919307    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:37.926177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.926154    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"]
Nov 01 20:21:37.930486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.930442    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\""
Nov 01 20:21:37.930486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.930474    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\""
Nov 01 20:21:37.930664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.930474    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-2fxjd\""
Nov 01 20:21:37.930664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.930485    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\""
Nov 01 20:21:37.930664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.930529    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\""
Nov 01 20:21:37.930900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.930878    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-b2b285556e64\""
Nov 01 20:21:37.993634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.993598    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:37.993864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.993799    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:37.993928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.993909    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:37.994067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:37.994051    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4n4\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.096013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.095916    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.096201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.096010    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.096201 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.096067    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found
Nov 01 20:21:38.096201 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.096093    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5: secret "agc-cred-b2b285556e64" not found
Nov 01 20:21:38.096201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.096163    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4n4\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.096201 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.096178    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials podName:41533593-7bb8-4b07-80fb-d97ec8637b41 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:38.596159 +0000 UTC m=+1344.011791451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5" (UID: "41533593-7bb8-4b07-80fb-d97ec8637b41") : secret "agc-cred-b2b285556e64" not found
Nov 01 20:21:38.096483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.096303    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.096794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.096773    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.096894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.096876    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.137972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.137941    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4n4\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.165198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.165163    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:38.169290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.169253    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"]
Nov 01 20:21:38.169858 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.169826    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5" podUID="41533593-7bb8-4b07-80fb-d97ec8637b41"
Nov 01 20:21:38.240361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.240328    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:38.410369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.410257    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.474670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.468378    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.609360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.609327    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume\") pod \"41533593-7bb8-4b07-80fb-d97ec8637b41\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") "
Nov 01 20:21:38.609534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.609373    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4\") pod \"41533593-7bb8-4b07-80fb-d97ec8637b41\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") "
Nov 01 20:21:38.609534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.609483    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume\") pod \"41533593-7bb8-4b07-80fb-d97ec8637b41\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") "
Nov 01 20:21:38.609761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.609736    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume" pod "41533593-7bb8-4b07-80fb-d97ec8637b41" (UID: "41533593-7bb8-4b07-80fb-d97ec8637b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:38.609898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.609874    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume" pod "41533593-7bb8-4b07-80fb-d97ec8637b41" (UID: "41533593-7bb8-4b07-80fb-d97ec8637b41"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:38.610201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.610183    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5\" (UID: \"41533593-7bb8-4b07-80fb-d97ec8637b41\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:38.610267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.610253    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:38.610304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.610274    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41533593-7bb8-4b07-80fb-d97ec8637b41-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:38.610337 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.610321    6479 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found
Nov 01 20:21:38.610337 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.610334    6479 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5: secret "agc-cred-b2b285556e64" not found
Nov 01 20:21:38.610405 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:38.610391    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials podName:41533593-7bb8-4b07-80fb-d97ec8637b41 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:39.610375258 +0000 UTC m=+1345.026007725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5" (UID: "41533593-7bb8-4b07-80fb-d97ec8637b41") : secret "agc-cred-b2b285556e64" not found
Nov 01 20:21:38.611907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.611876    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4" pod "41533593-7bb8-4b07-80fb-d97ec8637b41" (UID: "41533593-7bb8-4b07-80fb-d97ec8637b41"). InnerVolumeSpecName "kube-api-access-hn4n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:38.670335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.670240    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:38.711254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:38.711224    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4n4\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-kube-api-access-hn4n4\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:39.055617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:39.055537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:39.417460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:39.417383    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"
Nov 01 20:21:39.456113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:39.456076    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"]
Nov 01 20:21:39.481686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:39.479076    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-dc55wxrj5"]
Nov 01 20:21:39.628325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:39.628295    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41533593-7bb8-4b07-80fb-d97ec8637b41-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:39.939467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:39.939426    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:40.709950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:40.709921    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41533593-7bb8-4b07-80fb-d97ec8637b41" path="/var/lib/kubelet/pods/41533593-7bb8-4b07-80fb-d97ec8637b41/volumes"
Nov 01 20:21:40.973392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:40.973295    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:41.585106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:41.585073    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:42.530351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:42.530307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:43.338803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:43.338758    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:44.040915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:44.040876    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:44.679066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:44.679032    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:45.190408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:45.190377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:45.302109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:45.302069    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:45.386938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:45.386897    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:45.530733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:45.530697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:46.161883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:46.161844    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:46.801288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:46.801246    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:47.430499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:47.430451    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:48.172161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:48.172121    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:48.848589 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:48.848550    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:49.540287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:49.540246    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"]
Nov 01 20:21:49.550068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:49.550037    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"]
Nov 01 20:21:49.596830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:49.596788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:49.673131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:49.673081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:50.068087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:50.067940    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:50.569254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:50.569214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:51.170323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:51.170291    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:51.272516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:51.272475    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:51.373656 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:51.373618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:51.498671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:51.493199    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:21:51.576049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:51.576009    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:51.813512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:51.813424    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:52.287399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:52.287345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:53.216670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:53.216075    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:54.171841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:54.171801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:54.541271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:54.541229    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts" podUID="8880dea5-99d5-4512-ab04-744f461b1aec" containerName="payload-processing" containerID="cri-o://1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368" gracePeriod=25
Nov 01 20:21:54.550786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:54.550753    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq" podUID="e18d37c2-8f32-4933-84d9-691b54067543" containerName="payload-pre-processing" containerID="cri-o://713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d" gracePeriod=25
Nov 01 20:21:54.847394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:54.847372    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:21:54.884421 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:54.884292    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:21:54.943070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:54.943029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:55.009375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.009324    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7\" (UniqueName: \"kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7\") pod \"8880dea5-99d5-4512-ab04-744f461b1aec\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") "
Nov 01 20:21:55.009600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.009484    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume\") pod \"8880dea5-99d5-4512-ab04-744f461b1aec\" (UID: \"8880dea5-99d5-4512-ab04-744f461b1aec\") "
Nov 01 20:21:55.009722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.009697    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq\" (UniqueName: \"kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq\") pod \"e18d37c2-8f32-4933-84d9-691b54067543\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") "
Nov 01 20:21:55.011831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.010305    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume" pod "8880dea5-99d5-4512-ab04-744f461b1aec" (UID: "8880dea5-99d5-4512-ab04-744f461b1aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:55.012105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.012085    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume\") pod \"e18d37c2-8f32-4933-84d9-691b54067543\" (UID: \"e18d37c2-8f32-4933-84d9-691b54067543\") "
Nov 01 20:21:55.012763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.012623    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume" pod "e18d37c2-8f32-4933-84d9-691b54067543" (UID: "e18d37c2-8f32-4933-84d9-691b54067543"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:55.013503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.013481    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e18d37c2-8f32-4933-84d9-691b54067543-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:55.013616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.013603    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8880dea5-99d5-4512-ab04-744f461b1aec-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:55.045063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.045005    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7" pod "8880dea5-99d5-4512-ab04-744f461b1aec" (UID: "8880dea5-99d5-4512-ab04-744f461b1aec"). InnerVolumeSpecName "kube-api-access-dlhk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:55.065317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.065224    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq" pod "e18d37c2-8f32-4933-84d9-691b54067543" (UID: "e18d37c2-8f32-4933-84d9-691b54067543"). InnerVolumeSpecName "kube-api-access-vpvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:55.115291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.115251    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhk7\" (UniqueName: \"kubernetes.io/projected/8880dea5-99d5-4512-ab04-744f461b1aec-kube-api-access-dlhk7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:55.115291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.115295    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvzq\" (UniqueName: \"kubernetes.io/projected/e18d37c2-8f32-4933-84d9-691b54067543-kube-api-access-vpvzq\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:55.451185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.451143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:55.671670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.652084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:55.758857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.758763    6479 generic.go:356] "Generic (PLEG): container finished" podID="8880dea5-99d5-4512-ab04-744f461b1aec" containerID="1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368" exitCode=0
Nov 01 20:21:55.759304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.758998    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"
Nov 01 20:21:55.759782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.759018    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts" event={"ID":"8880dea5-99d5-4512-ab04-744f461b1aec","Type":"ContainerDied","Data":"1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368"}
Nov 01 20:21:55.759977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.759955    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts" event={"ID":"8880dea5-99d5-4512-ab04-744f461b1aec","Type":"ContainerDied","Data":"8c3b6b1d44851c852f3e63ffc001ef7bf9b48fcc71b6a9a24dd15c956222b7b1"}
Nov 01 20:21:55.760194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.760181    6479 scope.go:122] "RemoveContainer" containerID="1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368"
Nov 01 20:21:55.776007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.775972    6479 generic.go:356] "Generic (PLEG): container finished" podID="e18d37c2-8f32-4933-84d9-691b54067543" containerID="713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d" exitCode=0
Nov 01 20:21:55.778707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.776257    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq" event={"ID":"e18d37c2-8f32-4933-84d9-691b54067543","Type":"ContainerDied","Data":"713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d"}
Nov 01 20:21:55.778938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.778911    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq" event={"ID":"e18d37c2-8f32-4933-84d9-691b54067543","Type":"ContainerDied","Data":"3f3f89853c63ed025a0863c53f14c2218e44ed8b90d4f07a7c697b94bba540bb"}
Nov 01 20:21:55.779069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.776841    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"
Nov 01 20:21:55.826956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.826925    6479 scope.go:122] "RemoveContainer" containerID="1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368"
Nov 01 20:21:55.827741 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:55.827701    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368\": container with ID starting with 1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368 not found: ID does not exist" containerID="1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368"
Nov 01 20:21:55.827988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.827946    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368"} err="failed to get container status \"1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368\": rpc error: code = NotFound desc = could not find container \"1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368\": container with ID starting with 1ccba67d0bbe459d944d6b847c7bb5b8b352403684a65853d089336c41af2368 not found: ID does not exist"
Nov 01 20:21:55.828134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.828117    6479 scope.go:122] "RemoveContainer" containerID="713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d"
Nov 01 20:21:55.851575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.851523    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"]
Nov 01 20:21:55.855853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.855614    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:55.866477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.866446    6479 scope.go:122] "RemoveContainer" containerID="713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d"
Nov 01 20:21:55.867149 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:55.867120    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d\": container with ID starting with 713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d not found: ID does not exist" containerID="713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d"
Nov 01 20:21:55.867265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.867162    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d"} err="failed to get container status \"713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d\": rpc error: code = NotFound desc = could not find container \"713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d\": container with ID starting with 713bac43513f13bb62bfe3e5f2dacf2a79f1c755cb20552465f0dd9f7d90c20d not found: ID does not exist"
Nov 01 20:21:55.872159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.872136    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-933245-6d9bf7cbbf-7lxts"]
Nov 01 20:21:55.883988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.883723    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"]
Nov 01 20:21:55.887542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.887514    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-933245-5cd96d5679-jzbhq"]
Nov 01 20:21:55.978662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:55.978363    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:56.250181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:56.250149    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:56.574987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:56.574896    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:56.769671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:56.760565    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8880dea5-99d5-4512-ab04-744f461b1aec" path="/var/lib/kubelet/pods/8880dea5-99d5-4512-ab04-744f461b1aec/volumes"
Nov 01 20:21:56.771023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:56.770990    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18d37c2-8f32-4933-84d9-691b54067543" path="/var/lib/kubelet/pods/e18d37c2-8f32-4933-84d9-691b54067543/volumes"
Nov 01 20:21:56.890899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:56.890801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:57.154822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:57.154622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:57.247639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:57.247601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:57.433356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:57.433275    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"]
Nov 01 20:21:57.433830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:57.433793    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy" containerID="cri-o://d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d" gracePeriod=30
Nov 01 20:21:57.752997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:57.752963    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:57.893769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:57.893731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:58.042582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:58.042445    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:58.584418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:58.584387    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:59.173292 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:21:59.173253    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:21:59.366792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:59.366738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:21:59.966935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:21:59.966900    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:00.849437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:00.849405    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:01.546930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:01.546895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:01.646219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:01.646170    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:02.546114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.546074    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:02.744287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.744148    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:22:02.777295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.776395    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:02.853155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853113    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.853385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853173    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.853385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853269    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.853385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853319    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853625    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853708    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853770    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853788    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853802    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.853908    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs\") pod \"8af719b3-4af3-4cd2-86cd-d23723c7660b\" (UID: \"8af719b3-4af3-4cd2-86cd-d23723c7660b\") "
Nov 01 20:22:02.855167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.854928    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.855581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.855205    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.855581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.855312    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:22:02.855581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.855136    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.855822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.855797    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.858420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.858342    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:22:02.858872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.858833    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "kube-api-access-jcmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:22:02.859832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.859806    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:22:02.861089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.861056    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy" pod "8af719b3-4af3-4cd2-86cd-d23723c7660b" (UID: "8af719b3-4af3-4cd2-86cd-d23723c7660b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.888957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.888906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:02.902587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.902560    6479 generic.go:356] "Generic (PLEG): container finished" podID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerID="d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d" exitCode=0
Nov 01 20:22:02.902825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.902617    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"
Nov 01 20:22:02.902825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.902615    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" event={"ID":"8af719b3-4af3-4cd2-86cd-d23723c7660b","Type":"ContainerDied","Data":"d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d"}
Nov 01 20:22:02.902825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.902699    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7" event={"ID":"8af719b3-4af3-4cd2-86cd-d23723c7660b","Type":"ContainerDied","Data":"be5292098d6031ca1be5263b70fa64abbe9cb8ae3e1c04fcc6969db90bdead42"}
Nov 01 20:22:02.902825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.902733    6479 scope.go:122] "RemoveContainer" containerID="d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d"
Nov 01 20:22:02.917657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.917626    6479 scope.go:122] "RemoveContainer" containerID="d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d"
Nov 01 20:22:02.917985 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:22:02.917964    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d\": container with ID starting with d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d not found: ID does not exist" containerID="d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d"
Nov 01 20:22:02.918062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.917992    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d"} err="failed to get container status \"d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d\": rpc error: code = NotFound desc = could not find container \"d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d\": container with ID starting with d05851ac34bffd9884c9a9d124af0d965424546973af14252746b352cb33836d not found: ID does not exist"
Nov 01 20:22:02.927520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.927493    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"]
Nov 01 20:22:02.933573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.933551    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-kmhr7"]
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977015    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977062    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977080    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af719b3-4af3-4cd2-86cd-d23723c7660b-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977094    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmc8\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-kube-api-access-jcmc8\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977110    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977123    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977137    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.977667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:02.977150    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af719b3-4af3-4cd2-86cd-d23723c7660b-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:03.452053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:03.452010    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:04.111686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:04.107973    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:04.710965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:04.710912    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" path="/var/lib/kubelet/pods/8af719b3-4af3-4cd2-86cd-d23723c7660b/volumes"
Nov 01 20:22:04.875111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:04.875080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:05.640269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:05.640236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:06.334641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:06.334604    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:07.090292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:07.090243    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:07.824344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:07.824312    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:08.638726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:08.638697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:09.489446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:09.489407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:10.186322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:10.186283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:10.978227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:10.978188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:11.782565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:11.782526    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:12.445862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:12.445826    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:13.187162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.187128    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"]
Nov 01 20:22:13.188030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188008    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy"
Nov 01 20:22:13.188153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188030    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy"
Nov 01 20:22:13.188153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188074    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e18d37c2-8f32-4933-84d9-691b54067543" containerName="payload-pre-processing"
Nov 01 20:22:13.188153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188082    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18d37c2-8f32-4933-84d9-691b54067543" containerName="payload-pre-processing"
Nov 01 20:22:13.188153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188124    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8880dea5-99d5-4512-ab04-744f461b1aec" containerName="payload-processing"
Nov 01 20:22:13.188153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188132    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8880dea5-99d5-4512-ab04-744f461b1aec" containerName="payload-processing"
Nov 01 20:22:13.188402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188326    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8880dea5-99d5-4512-ab04-744f461b1aec" containerName="payload-processing"
Nov 01 20:22:13.188402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188345    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af719b3-4af3-4cd2-86cd-d23723c7660b" containerName="istio-proxy"
Nov 01 20:22:13.188402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.188362    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18d37c2-8f32-4933-84d9-691b54067543" containerName="payload-pre-processing"
Nov 01 20:22:13.193956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.193932    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.199152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.199130    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-57d281-serving-cert\""
Nov 01 20:22:13.212132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.212107    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"]
Nov 01 20:22:13.239911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.239883    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:13.240178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.240153    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.240471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.240452    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.240558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.240501    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.240558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.240542    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tck\" (UniqueName: \"kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.340427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.340393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:13.341995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.341973    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.342092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.342018    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.342092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.342040    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tck\" (UniqueName: \"kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.342092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.342066    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.342693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.342672    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.344623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.344601    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.344744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.344633    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.349995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.349973    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tck\" (UniqueName: \"kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck\") pod \"maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.444355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.444307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:13.521803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.521766    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:13.824997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.824964    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"]
Nov 01 20:22:13.826806 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:22:13.826777    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b5e420_b1d2_4d2c_9b9e_3c3be5605991.slice/crio-45f69bc99d89c57e6418c2554e08fbb2f899fb770cc49aef22dee1c5ebebb73c WatchSource:0}: Error finding container 45f69bc99d89c57e6418c2554e08fbb2f899fb770cc49aef22dee1c5ebebb73c: Status 404 returned error can't find the container with id 45f69bc99d89c57e6418c2554e08fbb2f899fb770cc49aef22dee1c5ebebb73c
Nov 01 20:22:13.828298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:13.828274    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:22:14.110719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:14.110082    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4" event={"ID":"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991","Type":"ContainerStarted","Data":"45f69bc99d89c57e6418c2554e08fbb2f899fb770cc49aef22dee1c5ebebb73c"}
Nov 01 20:22:14.519215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:14.519185    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:15.118708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:15.118672    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4" event={"ID":"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991","Type":"ContainerStarted","Data":"f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2"}
Nov 01 20:22:15.118902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:15.118801    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:15.137802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:15.137747    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4" podStartSLOduration=1.593647937 podStartE2EDuration="2.13773459s" totalImagesPullingTime="544.086653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:15.133708014 +0000 UTC m=+1380.549340488" watchObservedRunningTime="2026-11-01 20:22:15.13773459 +0000 UTC m=+1380.553367063"
Nov 01 20:22:15.274367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:15.274333    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:15.865477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:15.865436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:16.462265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:16.462233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:17.059588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:17.059551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:17.725321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:17.725278    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:17.917348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:17.917312    6479 scope.go:122] "RemoveContainer" containerID="72079bcb40c3e65d9dbc0a2fb7c3a8542c559fac940a02fe1a1664755431dcce"
Nov 01 20:22:17.928621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:17.928601    6479 scope.go:122] "RemoveContainer" containerID="d255956879cae64dde4df102c2cbd3e3bcc64dce0836eed88ed222d3ccb560ef"
Nov 01 20:22:18.324992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:18.324904    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:18.937585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:18.937546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:19.583248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:19.583214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:20.164009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:20.163964    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:20.811911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:20.811869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:21.132323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:21.132244    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:22:21.455719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:21.455685    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:22.101241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:22.101196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:22.541848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:22.541812    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:22.849538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:22.849462    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:23.495608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:23.495567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:24.147085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:24.147038    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:24.845869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:24.845839    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:25.627018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:25.626985    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:26.299011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:26.298977    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:26.331211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:26.331185    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:22:26.618039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:26.617955    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:27.049132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:27.049101    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:27.848324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:27.848289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:28.285418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:28.285385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:28.620337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:28.620226    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:29.171237 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:22:29.171199    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:22:29.255604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:29.255574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:29.847685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:29.847634    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:30.566031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:30.565996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:31.213595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:31.213554    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:31.900230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:31.900170    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:32.565940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:32.565905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:33.295889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:33.295848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:33.961513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:33.961477    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:34.743879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:34.743847    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:35.169284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:35.169187    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:35.543206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:35.543172    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:35.648780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:35.648732    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:35.743901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:35.743864    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:36.148238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:36.148200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:36.290574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:36.290537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:36.475707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:36.469996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:37.251126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:37.251098    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:38.062277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:38.062243    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:38.655251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:38.655213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:39.552771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:39.552735    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:39.993785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:39.993746    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:40.095034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:40.094996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:40.656601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:40.656559    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:40.785295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:40.785253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:41.268801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.268765    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:41.348991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.348948    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"]
Nov 01 20:22:41.354781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.354719    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.363952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.363926    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2babaef6-dockercfg-mqjk7\""
Nov 01 20:22:41.366079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.366054    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"]
Nov 01 20:22:41.371706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.370876    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2babaef6\""
Nov 01 20:22:41.398402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.398356    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:41.400252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.400092    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"]
Nov 01 20:22:41.410549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.410523    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.411025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.410985    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.411807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.411782    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzl24\" (UniqueName: \"kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.412046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.412027    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"]
Nov 01 20:22:41.603717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.516931    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lz8\" (UniqueName: \"kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-kvt29\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.603717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.517038    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.603717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.517138    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzl24\" (UniqueName: \"kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.603717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.517212    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-kvt29\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.603717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.518196    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.619518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.619465    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-kvt29\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.620143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.620117    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lz8\" (UniqueName: \"kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-kvt29\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.621593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.621568    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-kvt29\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.667552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.667522    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lz8\" (UniqueName: \"kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-kvt29\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.678564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.678327    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzl24\" (UniqueName: \"kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.755785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.755749    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:41.849013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.848685    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:41.911893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:41.911835    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:42.074230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.074198    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"]
Nov 01 20:22:42.076340 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:22:42.076316    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7110dd3_c906_47ae_b79a_98939a9e0d52.slice/crio-a91a416135fb29d766a6d59c5268e23bbb07ab6d2de556b44a4566572d217c2a WatchSource:0}: Error finding container a91a416135fb29d766a6d59c5268e23bbb07ab6d2de556b44a4566572d217c2a: Status 404 returned error can't find the container with id a91a416135fb29d766a6d59c5268e23bbb07ab6d2de556b44a4566572d217c2a
Nov 01 20:22:42.186332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.186305    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"]
Nov 01 20:22:42.187207 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:22:42.187177    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6e18e8_ada2_4755_be98_6871e43c8d43.slice/crio-89cc98bbd6e97b0afdacaa54e2a57aaa57c08603dac40c78f74bd5b7988c1d4b WatchSource:0}: Error finding container 89cc98bbd6e97b0afdacaa54e2a57aaa57c08603dac40c78f74bd5b7988c1d4b: Status 404 returned error can't find the container with id 89cc98bbd6e97b0afdacaa54e2a57aaa57c08603dac40c78f74bd5b7988c1d4b
Nov 01 20:22:42.631005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.630967    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" event={"ID":"cf6e18e8-ada2-4755-be98-6871e43c8d43","Type":"ContainerStarted","Data":"40780a4d5f6e40dd73a129a49797f408b1d76890acadfc812f24174db40a9c3a"}
Nov 01 20:22:42.631512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.631020    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" event={"ID":"cf6e18e8-ada2-4755-be98-6871e43c8d43","Type":"ContainerStarted","Data":"89cc98bbd6e97b0afdacaa54e2a57aaa57c08603dac40c78f74bd5b7988c1d4b"}
Nov 01 20:22:42.631512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.631060    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:42.632866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.632845    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" event={"ID":"e7110dd3-c906-47ae-b79a-98939a9e0d52","Type":"ContainerStarted","Data":"8e830b254402312625d6b9a149f42f19a87f76e96f8b218612f2b5e7217cc0fe"}
Nov 01 20:22:42.632986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.632888    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" event={"ID":"e7110dd3-c906-47ae-b79a-98939a9e0d52","Type":"ContainerStarted","Data":"a91a416135fb29d766a6d59c5268e23bbb07ab6d2de556b44a4566572d217c2a"}
Nov 01 20:22:42.632986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.632939    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:42.648256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.648214    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" podStartSLOduration=1.648203249 podStartE2EDuration="1.648203249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:42.647162185 +0000 UTC m=+1408.062794656" watchObservedRunningTime="2026-11-01 20:22:42.648203249 +0000 UTC m=+1408.063835779"
Nov 01 20:22:42.655079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.655058    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:42.664575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:42.664537    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" podStartSLOduration=1.6645251650000001 podStartE2EDuration="1.664525165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:42.661892124 +0000 UTC m=+1408.077524596" watchObservedRunningTime="2026-11-01 20:22:42.664525165 +0000 UTC m=+1408.080157637"
Nov 01 20:22:43.333991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:43.333962    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:43.973566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:43.973516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:44.689821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:44.689785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:45.388499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:45.388465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:46.136536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:46.136479    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:46.742611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:46.742568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:47.336949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:47.336905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:47.958588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:47.958544    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:48.573130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:48.573088    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:49.300675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:49.300584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:49.939901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:49.939850    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:50.537581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:50.537535    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:51.240538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:51.240501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:51.986744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:51.986705    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:52.065563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:52.065531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:52.148711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:52.142043    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:52.841578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:52.841529    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:53.637044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:53.637005    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:22:53.644358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:53.644330    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:22:53.644500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:53.644409    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:22:53.704576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:53.704537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:54.380616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:54.380571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:55.372344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:55.372304    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:56.052482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:56.052432    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:56.855380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:56.855342    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:57.524858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:57.524819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:58.328269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:58.328227    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:58.948142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:58.948108    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:22:59.171869 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:22:59.171826    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:22:59.615408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:22:59.615365    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:00.203638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:00.203590    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:00.847356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:00.847308    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:01.557271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:01.557228    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:02.227997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:02.227965    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:02.825443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:02.825411    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:03.459326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:03.459276    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:04.158204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:04.158165    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:04.823735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:04.823687    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:05.682101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:05.682049    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:06.274869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:06.274836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:06.857434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:06.857394    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:07.635363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:07.635319    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:08.256033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:08.255985    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:08.838833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:08.838791    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:09.456427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:09.456391    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:10.164879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:10.164840    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:10.265940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:10.265902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:10.950815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:10.950785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:11.635494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:11.635457    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:11.836999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:11.836955    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:12.450303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:12.450264    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:13.058510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:13.058474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:13.664161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:13.662966    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:14.559006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:14.558969    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:15.076274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:15.076238    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:15.386799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:15.386659    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:15.573870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:15.573836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:16.118866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:16.118807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:16.804584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:16.804546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:16.904723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:16.904688    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:17.008128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:17.008098    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:17.768416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:17.768383    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:18.166444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:18.166348    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:18.657241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:18.657210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:19.425558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:19.425509    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:20.068232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:20.068194    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:20.954441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:20.954401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:21.688988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:21.688940    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:22.334487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:22.334449    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:22.605966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:22.605880    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:23.027813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:23.027775    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:23.651260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:23.651228    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:23.757285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:23.757249    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:24.247214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:24.247180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:24.901248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:24.901166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:25.550618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:25.550588    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:26.260158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:26.260120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:26.968672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:26.968607    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:27.839432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:27.839385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:27.956674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:27.941991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:28.800195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:28.800161    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:29.143404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.143317    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:23:29.172823 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:23:29.172784    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:23:29.324815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.324766    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"]
Nov 01 20:23:29.325307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.325273    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4" podUID="f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" containerName="maas-api" containerID="cri-o://f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2" gracePeriod=30
Nov 01 20:23:29.367762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.366769    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:29.770675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.770420    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:29.812619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.812595    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:23:29.844766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.844735    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca\") pod \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") "
Nov 01 20:23:29.844934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.844919    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs\") pod \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") "
Nov 01 20:23:29.845016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.845003    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck\" (UniqueName: \"kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck\") pod \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") "
Nov 01 20:23:29.845085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.845045    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls\") pod \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\" (UID: \"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991\") "
Nov 01 20:23:29.845184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.845155    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca" pod "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" (UID: "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:29.845638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.845615    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:29.847447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.847419    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs" pod "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" (UID: "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:29.850287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.850248    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls" pod "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" (UID: "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:29.850417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.850385    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck" pod "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" (UID: "f1b5e420-b1d2-4d2c-9b9e-3c3be5605991"). InnerVolumeSpecName "kube-api-access-c6tck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:29.946767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.946718    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tck\" (UniqueName: \"kubernetes.io/projected/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-kube-api-access-c6tck\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:29.946767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.946758    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:29.946767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:29.946771    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:30.145017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.144925    6479 generic.go:356] "Generic (PLEG): container finished" podID="f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" containerID="f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2" exitCode=0
Nov 01 20:23:30.145017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.144973    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"
Nov 01 20:23:30.145567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.145006    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4" event={"ID":"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991","Type":"ContainerDied","Data":"f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2"}
Nov 01 20:23:30.145567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.145085    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4" event={"ID":"f1b5e420-b1d2-4d2c-9b9e-3c3be5605991","Type":"ContainerDied","Data":"45f69bc99d89c57e6418c2554e08fbb2f899fb770cc49aef22dee1c5ebebb73c"}
Nov 01 20:23:30.145567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.145108    6479 scope.go:122] "RemoveContainer" containerID="f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2"
Nov 01 20:23:30.159709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.159688    6479 scope.go:122] "RemoveContainer" containerID="f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2"
Nov 01 20:23:30.160067 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:23:30.160037    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2\": container with ID starting with f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2 not found: ID does not exist" containerID="f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2"
Nov 01 20:23:30.160170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.160077    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2"} err="failed to get container status \"f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2\": rpc error: code = NotFound desc = could not find container \"f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2\": container with ID starting with f1f802e9a5650b31cf32bc9e6c93a34f4e913c42b68a1d41632b965e6510aff2 not found: ID does not exist"
Nov 01 20:23:30.171609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.171580    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"]
Nov 01 20:23:30.175931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.175905    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57d281-6b69dfb779-zs6w4"]
Nov 01 20:23:30.712994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.712958    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" path="/var/lib/kubelet/pods/f1b5e420-b1d2-4d2c-9b9e-3c3be5605991/volumes"
Nov 01 20:23:30.888302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:30.888269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:31.848149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:31.848113    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:32.629266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:32.629211    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:33.334533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:33.334478    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:33.926564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:33.926519    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:34.063704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:34.063634    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:34.656052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:34.656020    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:35.344585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:35.344553    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:36.246059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:36.246026    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:36.816192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:36.816160    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:37.430244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:37.430207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:38.138604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:38.138556    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:38.769554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:38.769522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:38.874690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:38.874570    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:39.230622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:39.230577    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:39.613306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:39.613201    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:39.955481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:39.955439    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:40.194881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:40.194840    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:40.573269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:40.573215    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:41.212883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:41.212845    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:41.311273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:41.311241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:41.850342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:41.850305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:42.252176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:42.252144    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:42.468791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:42.468754    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:42.933190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:42.933151    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:43.141366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:43.141328    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:43.240228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:43.240189    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:43.402468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:43.402434    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:43.658443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:43.658365    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:44.243506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.243476    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"]
Nov 01 20:23:44.244847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.244823    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" containerName="maas-api"
Nov 01 20:23:44.244989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.244976    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" containerName="maas-api"
Nov 01 20:23:44.245302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.245290    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b5e420-b1d2-4d2c-9b9e-3c3be5605991" containerName="maas-api"
Nov 01 20:23:44.251622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.251595    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"]
Nov 01 20:23:44.251839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.251722    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.256366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.256343    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"e2e-conflict-b-1908b3f5-kserve-self-signed-certs\""
Nov 01 20:23:44.274407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.274368    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:44.405987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.388761    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.405987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.388823    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxbjr\" (UniqueName: \"kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.405987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.388945    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.405987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.388989    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.405987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.389030    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.405987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.389058    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.424667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.409225    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:44.491668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.491552    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.491668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.491622    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.491928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.491687    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.491928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.491717    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.492044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.491948    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.492044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.491989    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxbjr\" (UniqueName: \"kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.492160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.492139    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.492584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.492416    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.492584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.492541    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.515721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.495406    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.515721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.495975    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.527528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.524832    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxbjr\" (UniqueName: \"kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr\") pod \"e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.570684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.569186    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:44.862810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:44.862778    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"]
Nov 01 20:23:44.864690 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:23:44.864658    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e40533b_99d5_4c9e_8712_9b7914960a62.slice/crio-91be17f4534a7fa7ca7ef21ae6d1686e1a71332ff6139cefe5a2891005173e39 WatchSource:0}: Error finding container 91be17f4534a7fa7ca7ef21ae6d1686e1a71332ff6139cefe5a2891005173e39: Status 404 returned error can't find the container with id 91be17f4534a7fa7ca7ef21ae6d1686e1a71332ff6139cefe5a2891005173e39
Nov 01 20:23:45.289030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:45.288985    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7" event={"ID":"8e40533b-99d5-4c9e-8712-9b7914960a62","Type":"ContainerStarted","Data":"bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f"}
Nov 01 20:23:45.289246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:45.289042    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7" event={"ID":"8e40533b-99d5-4c9e-8712-9b7914960a62","Type":"ContainerStarted","Data":"91be17f4534a7fa7ca7ef21ae6d1686e1a71332ff6139cefe5a2891005173e39"}
Nov 01 20:23:45.311537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:45.311481    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7" podStartSLOduration=1.311466212 podStartE2EDuration="1.311466212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:45.308021366 +0000 UTC m=+1470.723653840" watchObservedRunningTime="2026-11-01 20:23:45.311466212 +0000 UTC m=+1470.727098684"
Nov 01 20:23:45.312877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:45.312845    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:45.569995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:45.569915    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:46.130492 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:46.130449    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:46.839700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:46.839664    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:47.499000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:47.498959    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:48.168512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:48.168469    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:48.755689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:48.755637    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:49.358891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:49.358846    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:49.954054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:49.954015    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:50.567840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:50.567807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:51.159783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:51.159731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:51.756001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:51.755962    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:52.430673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:52.430614    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:53.030193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:53.030160    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:53.896358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:53.896310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:54.570675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:54.570625    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:54.608144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:54.608088    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:54.626452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:54.626406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:55.029680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:55.029203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:55.432786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:55.432709    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:23:56.079597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:56.079560    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:56.793281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:56.793247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:56.893968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:56.893933    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:57.480838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:57.480797    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:58.124050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:58.124012    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:58.684670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:58.684571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:59.174068 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:23:59.174024    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:23:59.263941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:59.263903    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:23:59.861821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:23:59.861784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:00.049631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:00.049590    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"]
Nov 01 20:24:00.075502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:00.075461    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:00.303389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:00.303340    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:00.450391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:00.450347    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:00.820829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:00.820799    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:00.920484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:00.920450    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:01.020699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:01.020664    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:01.562635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:01.562602    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:02.300427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:02.300388    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:02.955463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:02.955420    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:03.050833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:03.050798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:03.663535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:03.663500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:03.766909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:03.764929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:04.269099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:04.269060    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:04.968077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:04.968036    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:05.749111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:05.749077    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:05.947569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:05.947534    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:06.527256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:06.527222    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:07.201423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:07.201388    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:07.867557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:07.867526    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:08.560277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:08.560233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:09.260435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:09.260396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:09.560861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:09.557851    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"]
Nov 01 20:24:09.576113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:09.576074    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"]
Nov 01 20:24:09.636691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:09.631364    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:09.732678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:09.732633    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:09.998209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:09.998154    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:10.258368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:10.258271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:10.993417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:10.993379    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:11.547207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:11.547169    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:12.133745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:12.133708    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:12.603953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:12.603912    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"]
Nov 01 20:24:12.604207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:12.604179    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl" podUID="9cc72527-0c94-42e7-a7fb-a62b7bfc2440" containerName="maas-api" containerID="cri-o://c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68" gracePeriod=30
Nov 01 20:24:12.633528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:12.633435    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:12.919207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:12.919115    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:13.211896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.211869    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:24:13.315298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.315265    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca\") pod \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") "
Nov 01 20:24:13.315468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.315369    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7\" (UniqueName: \"kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7\") pod \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") "
Nov 01 20:24:13.315529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.315506    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls\") pod \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") "
Nov 01 20:24:13.315733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.315715    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs\") pod \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\" (UID: \"9cc72527-0c94-42e7-a7fb-a62b7bfc2440\") "
Nov 01 20:24:13.315833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.315718    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca" pod "9cc72527-0c94-42e7-a7fb-a62b7bfc2440" (UID: "9cc72527-0c94-42e7-a7fb-a62b7bfc2440"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:13.316254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.316234    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:13.317691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.317633    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs" pod "9cc72527-0c94-42e7-a7fb-a62b7bfc2440" (UID: "9cc72527-0c94-42e7-a7fb-a62b7bfc2440"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:13.317815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.317790    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls" pod "9cc72527-0c94-42e7-a7fb-a62b7bfc2440" (UID: "9cc72527-0c94-42e7-a7fb-a62b7bfc2440"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:13.317916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.317890    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7" pod "9cc72527-0c94-42e7-a7fb-a62b7bfc2440" (UID: "9cc72527-0c94-42e7-a7fb-a62b7bfc2440"). InnerVolumeSpecName "kube-api-access-h8kk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:13.417642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.417607    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kk7\" (UniqueName: \"kubernetes.io/projected/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-kube-api-access-h8kk7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:13.417860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.417640    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:13.417860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.417687    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc72527-0c94-42e7-a7fb-a62b7bfc2440-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:13.646673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.641955    6479 generic.go:356] "Generic (PLEG): container finished" podID="9cc72527-0c94-42e7-a7fb-a62b7bfc2440" containerID="c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68" exitCode=0
Nov 01 20:24:13.646673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.642169    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl" event={"ID":"9cc72527-0c94-42e7-a7fb-a62b7bfc2440","Type":"ContainerDied","Data":"c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68"}
Nov 01 20:24:13.646673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.642223    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl" event={"ID":"9cc72527-0c94-42e7-a7fb-a62b7bfc2440","Type":"ContainerDied","Data":"d3a1dd058f90cad029c0a705f0e0b9e75b4c5795b2cdf19e934c8cbe4dcc9a6f"}
Nov 01 20:24:13.646673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.642253    6479 scope.go:122] "RemoveContainer" containerID="c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68"
Nov 01 20:24:13.646673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.642439    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"
Nov 01 20:24:13.654314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.649279    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:13.691116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.676628    6479 scope.go:122] "RemoveContainer" containerID="c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68"
Nov 01 20:24:13.691116 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:13.678125    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68\": container with ID starting with c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68 not found: ID does not exist" containerID="c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68"
Nov 01 20:24:13.691116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.678162    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68"} err="failed to get container status \"c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68\": rpc error: code = NotFound desc = could not find container \"c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68\": container with ID starting with c81d6770a8a3a9ff83a1a93746f3ed4e3e8cc270a56eeb88a563d8090809de68 not found: ID does not exist"
Nov 01 20:24:13.725830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.725785    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"]
Nov 01 20:24:13.729475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:13.729432    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaea41-57fbc885dd-xmvcl"]
Nov 01 20:24:14.164581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:14.164546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:14.558279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:14.558223    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerName="payload-processing" containerID="cri-o://8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5" gracePeriod=25
Nov 01 20:24:14.576938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:14.576876    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerName="payload-pre-processing" containerID="cri-o://5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4" gracePeriod=25
Nov 01 20:24:14.710086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:14.710052    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc72527-0c94-42e7-a7fb-a62b7bfc2440" path="/var/lib/kubelet/pods/9cc72527-0c94-42e7-a7fb-a62b7bfc2440/volumes"
Nov 01 20:24:14.866558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:14.866445    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:15.077415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.077358    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7" podUID="8e40533b-99d5-4c9e-8712-9b7914960a62" containerName="main" containerID="cri-o://bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f" gracePeriod=45
Nov 01 20:24:15.381453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.381427    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:24:15.458305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458263    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache\") pod \"8e40533b-99d5-4c9e-8712-9b7914960a62\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") "
Nov 01 20:24:15.458519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458385    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home\") pod \"8e40533b-99d5-4c9e-8712-9b7914960a62\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") "
Nov 01 20:24:15.458519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458416    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir\") pod \"8e40533b-99d5-4c9e-8712-9b7914960a62\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") "
Nov 01 20:24:15.458519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458493    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs\") pod \"8e40533b-99d5-4c9e-8712-9b7914960a62\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") "
Nov 01 20:24:15.458724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458519    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr\" (UniqueName: \"kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr\") pod \"8e40533b-99d5-4c9e-8712-9b7914960a62\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") "
Nov 01 20:24:15.458724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458597    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache" pod "8e40533b-99d5-4c9e-8712-9b7914960a62" (UID: "8e40533b-99d5-4c9e-8712-9b7914960a62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.458724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458704    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home" pod "8e40533b-99d5-4c9e-8712-9b7914960a62" (UID: "8e40533b-99d5-4c9e-8712-9b7914960a62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.458874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458765    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir" pod "8e40533b-99d5-4c9e-8712-9b7914960a62" (UID: "8e40533b-99d5-4c9e-8712-9b7914960a62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.458874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.458839    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm\") pod \"8e40533b-99d5-4c9e-8712-9b7914960a62\" (UID: \"8e40533b-99d5-4c9e-8712-9b7914960a62\") "
Nov 01 20:24:15.459729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.459699    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.459867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.459731    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.459867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.459748    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.460937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.460901    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm" pod "8e40533b-99d5-4c9e-8712-9b7914960a62" (UID: "8e40533b-99d5-4c9e-8712-9b7914960a62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.461044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.460937    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs" pod "8e40533b-99d5-4c9e-8712-9b7914960a62" (UID: "8e40533b-99d5-4c9e-8712-9b7914960a62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:15.461044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.460964    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr" pod "8e40533b-99d5-4c9e-8712-9b7914960a62" (UID: "8e40533b-99d5-4c9e-8712-9b7914960a62"). InnerVolumeSpecName "kube-api-access-xxbjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:15.561183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.561145    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40533b-99d5-4c9e-8712-9b7914960a62-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.561183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.561182    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxbjr\" (UniqueName: \"kubernetes.io/projected/8e40533b-99d5-4c9e-8712-9b7914960a62-kube-api-access-xxbjr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.561640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.561196    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40533b-99d5-4c9e-8712-9b7914960a62-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.583452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.583418    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:15.687716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.687680    6479 generic.go:356] "Generic (PLEG): container finished" podID="8e40533b-99d5-4c9e-8712-9b7914960a62" containerID="bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f" exitCode=0
Nov 01 20:24:15.688100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.688071    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7" event={"ID":"8e40533b-99d5-4c9e-8712-9b7914960a62","Type":"ContainerDied","Data":"bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f"}
Nov 01 20:24:15.688251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.688237    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7" event={"ID":"8e40533b-99d5-4c9e-8712-9b7914960a62","Type":"ContainerDied","Data":"91be17f4534a7fa7ca7ef21ae6d1686e1a71332ff6139cefe5a2891005173e39"}
Nov 01 20:24:15.688355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.688342    6479 scope.go:122] "RemoveContainer" containerID="bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f"
Nov 01 20:24:15.688598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.688576    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"
Nov 01 20:24:15.707464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.707425    6479 scope.go:122] "RemoveContainer" containerID="bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f"
Nov 01 20:24:15.707857 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:15.707832    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f\": container with ID starting with bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f not found: ID does not exist" containerID="bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f"
Nov 01 20:24:15.707960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.707869    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f"} err="failed to get container status \"bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f\": rpc error: code = NotFound desc = could not find container \"bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f\": container with ID starting with bfa79a06c32256b67900abb2c554bcdb7bc365e65c815ffe4d1445d8ebbe596f not found: ID does not exist"
Nov 01 20:24:15.721625 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.721552    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"]
Nov 01 20:24:15.723679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.723634    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:15.724816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:15.724794    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-b-1908b3f5-kserve-f9f8fdf77-vdzk7"]
Nov 01 20:24:16.078556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.078477    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:16.382585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.382495    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"]
Nov 01 20:24:16.383440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.383183    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" containerID="cri-o://f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a" gracePeriod=30
Nov 01 20:24:16.405061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.405023    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:16.711173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.711139    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e40533b-99d5-4c9e-8712-9b7914960a62" path="/var/lib/kubelet/pods/8e40533b-99d5-4c9e-8712-9b7914960a62/volumes"
Nov 01 20:24:16.846748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.846712    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:16.971098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:16.970954    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:24:17.162288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:17.162254    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:17.974670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:17.972170    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:18.969929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:18.969442    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:19.748057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:19.748004    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:20.354469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:20.354415    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:20.936150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:20.936116    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:20.961716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:20.961680    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:21.135576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.135536    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:24:21.135769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.135618    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:24:21.667857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.667798    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:24:21.681071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.681039    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.795738    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.795859    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.795950    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.795999    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.796030    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.796159    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.796400    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.796469    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.796537    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo\") pod \"52b2a924-7fde-427d-892d-cfaf07899030\" (UID: \"52b2a924-7fde-427d-892d-cfaf07899030\") "
Nov 01 20:24:21.818206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.816422    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.822706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.819665    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.822706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.822158    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.823026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.822998    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:21.823517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.823497    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:24:21.823725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.823684    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "kube-api-access-p8bxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:21.825074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.825034    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.825074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.825050    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:21.826631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.826581    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs" pod "52b2a924-7fde-427d-892d-cfaf07899030" (UID: "52b2a924-7fde-427d-892d-cfaf07899030"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.836359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.836328    6479 generic.go:356] "Generic (PLEG): container finished" podID="52b2a924-7fde-427d-892d-cfaf07899030" containerID="f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a" exitCode=0
Nov 01 20:24:21.836486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.836370    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" event={"ID":"52b2a924-7fde-427d-892d-cfaf07899030","Type":"ContainerDied","Data":"f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a"}
Nov 01 20:24:21.836486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.836420    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l" event={"ID":"52b2a924-7fde-427d-892d-cfaf07899030","Type":"ContainerDied","Data":"77fd51054a82ef6cf26b3d1d4f6b8560463548a8484d900c06ba704e733a1ccf"}
Nov 01 20:24:21.836486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.836448    6479 scope.go:122] "RemoveContainer" containerID="f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a"
Nov 01 20:24:21.836666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.836592    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"
Nov 01 20:24:21.859712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.859623    6479 scope.go:122] "RemoveContainer" containerID="309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"
Nov 01 20:24:21.873064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.873034    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"]
Nov 01 20:24:21.873206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.873188    6479 scope.go:122] "RemoveContainer" containerID="f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a"
Nov 01 20:24:21.873578 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:21.873550    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a\": container with ID starting with f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a not found: ID does not exist" containerID="f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a"
Nov 01 20:24:21.873715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.873591    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a"} err="failed to get container status \"f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a\": rpc error: code = NotFound desc = could not find container \"f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a\": container with ID starting with f3a28f9aedc08344996a73624f6137cbaecaf555f316e5b283abb920098f971a not found: ID does not exist"
Nov 01 20:24:21.873715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.873617    6479 scope.go:122] "RemoveContainer" containerID="309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"
Nov 01 20:24:21.874012 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:21.873985    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968\": container with ID starting with 309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968 not found: ID does not exist" containerID="309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"
Nov 01 20:24:21.874078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.874021    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968"} err="failed to get container status \"309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968\": rpc error: code = NotFound desc = could not find container \"309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968\": container with ID starting with 309acaf8b07868ba90325345d5e516f3b20b64bf0c8585079820f3f12cdd5968 not found: ID does not exist"
Nov 01 20:24:21.876787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.876751    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-hfs2l"]
Nov 01 20:24:21.925211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925136    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925178    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b2a924-7fde-427d-892d-cfaf07899030-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925189    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925199    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b2a924-7fde-427d-892d-cfaf07899030-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925209    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8bxh\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-kube-api-access-p8bxh\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925219    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b2a924-7fde-427d-892d-cfaf07899030-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925229    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925239    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.925491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.925248    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b2a924-7fde-427d-892d-cfaf07899030-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.989301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:21.989261    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:22.346681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:22.346638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:22.643134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:22.643051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:22.710615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:22.710585    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b2a924-7fde-427d-892d-cfaf07899030" path="/var/lib/kubelet/pods/52b2a924-7fde-427d-892d-cfaf07899030/volumes"
Nov 01 20:24:23.039030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:23.038995    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:23.360999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:23.360902    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"]
Nov 01 20:24:23.376322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:23.375627    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"]
Nov 01 20:24:23.425492 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:23.425448    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:23.522930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:23.522884    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:23.769666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:23.769372    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:24.114503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:24.114414    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:24.746120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:24.746086    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:25.343581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:25.343543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:26.038228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:26.038193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:26.641084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:26.641045    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:27.236546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.236508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:27.735763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.735731    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"]
Nov 01 20:24:27.736466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736451    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9cc72527-0c94-42e7-a7fb-a62b7bfc2440" containerName="maas-api"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736467    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc72527-0c94-42e7-a7fb-a62b7bfc2440" containerName="maas-api"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736477    6479 cpu_manager.go:414] "containerMap: removing container" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736482    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736490    6479 cpu_manager.go:414] "containerMap: removing container" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736495    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736523    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8e40533b-99d5-4c9e-8712-9b7914960a62" containerName="main"
Nov 01 20:24:27.736539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736528    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e40533b-99d5-4c9e-8712-9b7914960a62" containerName="main"
Nov 01 20:24:27.736791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736624    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e40533b-99d5-4c9e-8712-9b7914960a62" containerName="main"
Nov 01 20:24:27.736791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736638    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc72527-0c94-42e7-a7fb-a62b7bfc2440" containerName="maas-api"
Nov 01 20:24:27.736791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736660    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy"
Nov 01 20:24:27.736791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.736667    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b2a924-7fde-427d-892d-cfaf07899030" containerName="istio-proxy"
Nov 01 20:24:27.741702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.741683    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743175    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqkp\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743230    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743266    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743405    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743471    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743541    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743707    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743759    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.743867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.743835    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.749641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.749606    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"]
Nov 01 20:24:27.844825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.844780    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.844825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.844830    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.844856    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.844929    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.844958    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.844990    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845072    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqkp\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845122    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845172    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845335    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845400    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845613    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845630    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.845794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.845639    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.847507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.847490    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.847507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.847495    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.852955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.852926    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqkp\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.853078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.853026    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:27.874376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:27.874349    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:28.060979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.060894    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:28.345623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.345598    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"]
Nov 01 20:24:28.348005 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:24:28.347976    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30ec562_4f03_46d5_a99a_3e34973a3e8f.slice/crio-0744756e7f4ce2df44e5759af0f26a965ee26f69208af9f1fc26739bcce4c75d WatchSource:0}: Error finding container 0744756e7f4ce2df44e5759af0f26a965ee26f69208af9f1fc26739bcce4c75d: Status 404 returned error can't find the container with id 0744756e7f4ce2df44e5759af0f26a965ee26f69208af9f1fc26739bcce4c75d
Nov 01 20:24:28.352110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.352058    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:24:28.352216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.352147    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:24:28.352216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.352190    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:24:28.362894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.362861    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv" podUID="745458a2-aff2-4647-9933-ded4c77f8832" containerName="payload-processing" containerID="cri-o://9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446" gracePeriod=25
Nov 01 20:24:28.376417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.376385    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs" podUID="ffbe6911-ff8e-48a4-80be-00a9b5da73f9" containerName="payload-pre-processing" containerID="cri-o://59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82" gracePeriod=25
Nov 01 20:24:28.589685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.574960    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:28.743523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.743105    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:24:28.781134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.781098    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv\" (UniqueName: \"kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv\") pod \"745458a2-aff2-4647-9933-ded4c77f8832\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") "
Nov 01 20:24:28.781605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.781573    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume\") pod \"745458a2-aff2-4647-9933-ded4c77f8832\" (UID: \"745458a2-aff2-4647-9933-ded4c77f8832\") "
Nov 01 20:24:28.782837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.782812    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume" pod "745458a2-aff2-4647-9933-ded4c77f8832" (UID: "745458a2-aff2-4647-9933-ded4c77f8832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:28.786135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.786112    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv" pod "745458a2-aff2-4647-9933-ded4c77f8832" (UID: "745458a2-aff2-4647-9933-ded4c77f8832"). InnerVolumeSpecName "kube-api-access-d8zdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:28.875743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.875711    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:24:28.884070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.884041    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l\" (UniqueName: \"kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l\") pod \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") "
Nov 01 20:24:28.884207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.884098    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume\") pod \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\" (UID: \"ffbe6911-ff8e-48a4-80be-00a9b5da73f9\") "
Nov 01 20:24:28.884611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.884578    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume" pod "ffbe6911-ff8e-48a4-80be-00a9b5da73f9" (UID: "ffbe6911-ff8e-48a4-80be-00a9b5da73f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:28.884738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.884659    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zdv\" (UniqueName: \"kubernetes.io/projected/745458a2-aff2-4647-9933-ded4c77f8832-kube-api-access-d8zdv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:28.884738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.884683    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745458a2-aff2-4647-9933-ded4c77f8832-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:28.886415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.886390    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l" pod "ffbe6911-ff8e-48a4-80be-00a9b5da73f9" (UID: "ffbe6911-ff8e-48a4-80be-00a9b5da73f9"). InnerVolumeSpecName "kube-api-access-wsh9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:28.928165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.928131    6479 generic.go:356] "Generic (PLEG): container finished" podID="745458a2-aff2-4647-9933-ded4c77f8832" containerID="9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446" exitCode=0
Nov 01 20:24:28.928328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.928232    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv" event={"ID":"745458a2-aff2-4647-9933-ded4c77f8832","Type":"ContainerDied","Data":"9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446"}
Nov 01 20:24:28.928328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.928268    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv" event={"ID":"745458a2-aff2-4647-9933-ded4c77f8832","Type":"ContainerDied","Data":"f1b740d5ab2f60f8ceeea4aed80e9ad1de2695e97e79f84140abfad941fed891"}
Nov 01 20:24:28.928328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.928289    6479 scope.go:122] "RemoveContainer" containerID="9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446"
Nov 01 20:24:28.928537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.928415    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"
Nov 01 20:24:28.937617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.937576    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" event={"ID":"b30ec562-4f03-46d5-a99a-3e34973a3e8f","Type":"ContainerStarted","Data":"c95132d6627bffa0dce49f650bd6e35a730788988408ebd2e02d45d7548834d3"}
Nov 01 20:24:28.938153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.937703    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" event={"ID":"b30ec562-4f03-46d5-a99a-3e34973a3e8f","Type":"ContainerStarted","Data":"0744756e7f4ce2df44e5759af0f26a965ee26f69208af9f1fc26739bcce4c75d"}
Nov 01 20:24:28.941544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.940889    6479 generic.go:356] "Generic (PLEG): container finished" podID="ffbe6911-ff8e-48a4-80be-00a9b5da73f9" containerID="59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82" exitCode=0
Nov 01 20:24:28.941544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.940942    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs" event={"ID":"ffbe6911-ff8e-48a4-80be-00a9b5da73f9","Type":"ContainerDied","Data":"59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82"}
Nov 01 20:24:28.941544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.940970    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs" event={"ID":"ffbe6911-ff8e-48a4-80be-00a9b5da73f9","Type":"ContainerDied","Data":"c92a6769b5846231860163a8adab00befacd804ca93fabea5b12ed4c96c59aa6"}
Nov 01 20:24:28.941544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.941020    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"
Nov 01 20:24:28.956467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.956143    6479 scope.go:122] "RemoveContainer" containerID="9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446"
Nov 01 20:24:28.956602 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:28.956570    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446\": container with ID starting with 9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446 not found: ID does not exist" containerID="9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446"
Nov 01 20:24:28.956739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.956615    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446"} err="failed to get container status \"9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446\": rpc error: code = NotFound desc = could not find container \"9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446\": container with ID starting with 9e3a8846103d6525c9eb4aa696d50862c4ca29a12b1c69430653dd29646ca446 not found: ID does not exist"
Nov 01 20:24:28.956739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.956639    6479 scope.go:122] "RemoveContainer" containerID="59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82"
Nov 01 20:24:28.964839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.964756    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podStartSLOduration=1.964740612 podStartE2EDuration="1.964740612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:28.964258577 +0000 UTC m=+1514.379891046" watchObservedRunningTime="2026-11-01 20:24:28.964740612 +0000 UTC m=+1514.380373085"
Nov 01 20:24:28.983088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.981470    6479 scope.go:122] "RemoveContainer" containerID="59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82"
Nov 01 20:24:28.983088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.981589    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"]
Nov 01 20:24:28.983088 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:28.982010    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82\": container with ID starting with 59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82 not found: ID does not exist" containerID="59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82"
Nov 01 20:24:28.983088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.982044    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82"} err="failed to get container status \"59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82\": rpc error: code = NotFound desc = could not find container \"59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82\": container with ID starting with 59496fe0cab0abf696dcc78de2a69d03ed3e3ffa0515e4e4c21a68ca3f9f3c82 not found: ID does not exist"
Nov 01 20:24:28.986920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.986190    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:28.986920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.986225    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsh9l\" (UniqueName: \"kubernetes.io/projected/ffbe6911-ff8e-48a4-80be-00a9b5da73f9-kube-api-access-wsh9l\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:28.986920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.986295    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6f810f-6f84bb8b8c-68hwv"]
Nov 01 20:24:28.995504 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:28.995481    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"]
Nov 01 20:24:29.001424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:29.001398    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6f810f-5574bd88cf-297hs"]
Nov 01 20:24:29.061436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:29.061401    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:29.062701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:29.062631    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:29.062822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:29.062716    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:29.181785 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:29.181746    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:24:29.266145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:29.266003    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:29.772770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:29.772707    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:30.057162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.057029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:30.066697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.066198    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:30.066697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.066261    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:30.711307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.711275    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745458a2-aff2-4647-9933-ded4c77f8832" path="/var/lib/kubelet/pods/745458a2-aff2-4647-9933-ded4c77f8832/volumes"
Nov 01 20:24:30.711867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.711841    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffbe6911-ff8e-48a4-80be-00a9b5da73f9" path="/var/lib/kubelet/pods/ffbe6911-ff8e-48a4-80be-00a9b5da73f9/volumes"
Nov 01 20:24:30.748093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.748058    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:30.936059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:30.936018    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:31.069113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:31.069022    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:31.069388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:31.069361    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:31.398573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:31.397249    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:31.988510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:31.988470    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:32.069669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.068747    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:32.069669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.068809    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:32.128593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.128560    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:32.409830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.409751    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-63c56c"]
Nov 01 20:24:32.410414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.410400    6479 cpu_manager.go:414] "containerMap: removing container" podUID="ffbe6911-ff8e-48a4-80be-00a9b5da73f9" containerName="payload-pre-processing"
Nov 01 20:24:32.410464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.410416    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbe6911-ff8e-48a4-80be-00a9b5da73f9" containerName="payload-pre-processing"
Nov 01 20:24:32.410464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.410446    6479 cpu_manager.go:414] "containerMap: removing container" podUID="745458a2-aff2-4647-9933-ded4c77f8832" containerName="payload-processing"
Nov 01 20:24:32.410464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.410451    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745458a2-aff2-4647-9933-ded4c77f8832" containerName="payload-processing"
Nov 01 20:24:32.410565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.410556    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="745458a2-aff2-4647-9933-ded4c77f8832" containerName="payload-processing"
Nov 01 20:24:32.410600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.410573    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbe6911-ff8e-48a4-80be-00a9b5da73f9" containerName="payload-pre-processing"
Nov 01 20:24:32.419957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.416049    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:32.419957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.419777    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\""
Nov 01 20:24:32.421609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.421581    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-63c56c"]
Nov 01 20:24:32.449111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.449084    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dccjr\" (UniqueName: \"kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr\") pod \"test-curl-7417-63c56c\" (UID: \"8e42d602-3197-4d66-b8b4-7e22b22dc637\") " pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:32.550953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.550915    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dccjr\" (UniqueName: \"kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr\") pod \"test-curl-7417-63c56c\" (UID: \"8e42d602-3197-4d66-b8b4-7e22b22dc637\") " pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:32.579610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.579571    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dccjr\" (UniqueName: \"kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr\") pod \"test-curl-7417-63c56c\" (UID: \"8e42d602-3197-4d66-b8b4-7e22b22dc637\") " pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:32.733638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.733600    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:32.791243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:32.791213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:33.067222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.067140    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:33.067803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.067770    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:33.244598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.244570    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-63c56c"]
Nov 01 20:24:33.245510 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:24:33.245487    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e42d602_3197_4d66_b8b4_7e22b22dc637.slice/crio-3cbe79c6d3ba51e04d792d003816800f61347b2bda46f90f9e71b1012791ab98 WatchSource:0}: Error finding container 3cbe79c6d3ba51e04d792d003816800f61347b2bda46f90f9e71b1012791ab98: Status 404 returned error can't find the container with id 3cbe79c6d3ba51e04d792d003816800f61347b2bda46f90f9e71b1012791ab98
Nov 01 20:24:33.290046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.290014    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"]
Nov 01 20:24:33.292540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.292502    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8" podUID="c3953f2b-58ea-4e45-9308-9dedfba106c2" containerName="maas-api" containerID="cri-o://2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45" gracePeriod=30
Nov 01 20:24:33.331498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.331468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:33.586728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.577120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:33.947124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:33.947100    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:24:34.000991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.000951    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls\") pod \"c3953f2b-58ea-4e45-9308-9dedfba106c2\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") "
Nov 01 20:24:34.001181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.001146    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca\") pod \"c3953f2b-58ea-4e45-9308-9dedfba106c2\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") "
Nov 01 20:24:34.001277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.001259    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf\" (UniqueName: \"kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf\") pod \"c3953f2b-58ea-4e45-9308-9dedfba106c2\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") "
Nov 01 20:24:34.001396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.001382    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs\") pod \"c3953f2b-58ea-4e45-9308-9dedfba106c2\" (UID: \"c3953f2b-58ea-4e45-9308-9dedfba106c2\") "
Nov 01 20:24:34.003226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.003201    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca" pod "c3953f2b-58ea-4e45-9308-9dedfba106c2" (UID: "c3953f2b-58ea-4e45-9308-9dedfba106c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:34.007463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.005882    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs" pod "c3953f2b-58ea-4e45-9308-9dedfba106c2" (UID: "c3953f2b-58ea-4e45-9308-9dedfba106c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:34.007463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.005965    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls" pod "c3953f2b-58ea-4e45-9308-9dedfba106c2" (UID: "c3953f2b-58ea-4e45-9308-9dedfba106c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:34.007463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.006066    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf" pod "c3953f2b-58ea-4e45-9308-9dedfba106c2" (UID: "c3953f2b-58ea-4e45-9308-9dedfba106c2"). InnerVolumeSpecName "kube-api-access-g6fnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:34.014508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.014483    6479 generic.go:356] "Generic (PLEG): container finished" podID="c3953f2b-58ea-4e45-9308-9dedfba106c2" containerID="2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45" exitCode=0
Nov 01 20:24:34.014762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.014745    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8" event={"ID":"c3953f2b-58ea-4e45-9308-9dedfba106c2","Type":"ContainerDied","Data":"2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45"}
Nov 01 20:24:34.014835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.014823    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8" event={"ID":"c3953f2b-58ea-4e45-9308-9dedfba106c2","Type":"ContainerDied","Data":"cb05e333c74d4b32507a8efe0e6710fc325c263e8ba1f7913e393b5b5d68c2fe"}
Nov 01 20:24:34.014875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.014849    6479 scope.go:122] "RemoveContainer" containerID="2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45"
Nov 01 20:24:34.014957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.014948    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"
Nov 01 20:24:34.018707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.018683    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-63c56c" event={"ID":"8e42d602-3197-4d66-b8b4-7e22b22dc637","Type":"ContainerStarted","Data":"4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa"}
Nov 01 20:24:34.018803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.018720    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-63c56c" event={"ID":"8e42d602-3197-4d66-b8b4-7e22b22dc637","Type":"ContainerStarted","Data":"3cbe79c6d3ba51e04d792d003816800f61347b2bda46f90f9e71b1012791ab98"}
Nov 01 20:24:34.038115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.038091    6479 scope.go:122] "RemoveContainer" containerID="2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45"
Nov 01 20:24:34.038419 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:34.038393    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45\": container with ID starting with 2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45 not found: ID does not exist" containerID="2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45"
Nov 01 20:24:34.038483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.038430    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45"} err="failed to get container status \"2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45\": rpc error: code = NotFound desc = could not find container \"2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45\": container with ID starting with 2507e533af4bad8c2aa4f85af2048691ce882ef88b7c2ace0f115057ebc3de45 not found: ID does not exist"
Nov 01 20:24:34.040781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.040743    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-63c56c" podStartSLOduration=2.040731361 podStartE2EDuration="2.040731361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:34.036946046 +0000 UTC m=+1519.452578520" watchObservedRunningTime="2026-11-01 20:24:34.040731361 +0000 UTC m=+1519.456363834"
Nov 01 20:24:34.076695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.068731    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:34.076695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.068785    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:34.076695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.069117    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"]
Nov 01 20:24:34.094239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.094200    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6f810f-84b9bb98b9-2q6n8"]
Nov 01 20:24:34.103512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.103484    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:34.103512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.103511    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3953f2b-58ea-4e45-9308-9dedfba106c2-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:34.103705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.103522    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3953f2b-58ea-4e45-9308-9dedfba106c2-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:34.103705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.103531    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fnf\" (UniqueName: \"kubernetes.io/projected/c3953f2b-58ea-4e45-9308-9dedfba106c2-kube-api-access-g6fnf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:34.489066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.489029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:34.713301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.713270    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3953f2b-58ea-4e45-9308-9dedfba106c2" path="/var/lib/kubelet/pods/c3953f2b-58ea-4e45-9308-9dedfba106c2/volumes"
Nov 01 20:24:34.900823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.900771    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-63c56c"]
Nov 01 20:24:34.905337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.905304    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-63c56c"]
Nov 01 20:24:34.947064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.947036    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:24:34.999183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:34.999136    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.020707    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c\" (UniqueName: \"kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c\") pod \"bacf3d06-22d0-4637-bc28-991f58a055ad\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") "
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.020806    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv\" (UniqueName: \"kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv\") pod \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") "
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.020858    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume\") pod \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\" (UID: \"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1\") "
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.020895    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume\") pod \"bacf3d06-22d0-4637-bc28-991f58a055ad\" (UID: \"bacf3d06-22d0-4637-bc28-991f58a055ad\") "
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.021932    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume" pod "bacf3d06-22d0-4637-bc28-991f58a055ad" (UID: "bacf3d06-22d0-4637-bc28-991f58a055ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.023997    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume" pod "0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" (UID: "0a8f350c-b464-493d-8a6c-ef9bd7a9daa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.026683    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv" pod "0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" (UID: "0a8f350c-b464-493d-8a6c-ef9bd7a9daa1"). InnerVolumeSpecName "kube-api-access-lwzpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:35.027770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.027579    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c" pod "bacf3d06-22d0-4637-bc28-991f58a055ad" (UID: "bacf3d06-22d0-4637-bc28-991f58a055ad"). InnerVolumeSpecName "kube-api-access-dzg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:35.035827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.035796    6479 generic.go:356] "Generic (PLEG): container finished" podID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerID="5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4" exitCode=0
Nov 01 20:24:35.036003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.035850    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"
Nov 01 20:24:35.036003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.035886    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" event={"ID":"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1","Type":"ContainerDied","Data":"5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4"}
Nov 01 20:24:35.036003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.035952    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64" event={"ID":"0a8f350c-b464-493d-8a6c-ef9bd7a9daa1","Type":"ContainerDied","Data":"11febde413d26205fad70231d38188b7ca17493582777a0ff2e832384d475600"}
Nov 01 20:24:35.036003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.035984    6479 scope.go:122] "RemoveContainer" containerID="5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4"
Nov 01 20:24:35.038071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.038039    6479 generic.go:356] "Generic (PLEG): container finished" podID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerID="8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5" exitCode=0
Nov 01 20:24:35.038182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.038086    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"
Nov 01 20:24:35.038182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.038125    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" event={"ID":"bacf3d06-22d0-4637-bc28-991f58a055ad","Type":"ContainerDied","Data":"8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5"}
Nov 01 20:24:35.038290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.038180    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4" event={"ID":"bacf3d06-22d0-4637-bc28-991f58a055ad","Type":"ContainerDied","Data":"319a3eca7b5bfb7950b61b4a08d624136b8a46db1f193d4a136f58be12162787"}
Nov 01 20:24:35.052818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.052743    6479 scope.go:122] "RemoveContainer" containerID="5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4"
Nov 01 20:24:35.053239 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:35.053145    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4\": container with ID starting with 5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4 not found: ID does not exist" containerID="5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4"
Nov 01 20:24:35.053239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.053171    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4"} err="failed to get container status \"5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4\": rpc error: code = NotFound desc = could not find container \"5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4\": container with ID starting with 5cf69de1182ff998511e85d0afea6ba0969dbc4807e63ed510cc5f649793c6a4 not found: ID does not exist"
Nov 01 20:24:35.053239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.053186    6479 scope.go:122] "RemoveContainer" containerID="8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5"
Nov 01 20:24:35.062603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.062233    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:24:35.062603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.062279    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 01 20:24:35.070262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.069873    6479 scope.go:122] "RemoveContainer" containerID="8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5"
Nov 01 20:24:35.070376 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:35.070268    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5\": container with ID starting with 8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5 not found: ID does not exist" containerID="8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5"
Nov 01 20:24:35.070376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.070316    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5"} err="failed to get container status \"8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5\": rpc error: code = NotFound desc = could not find container \"8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5\": container with ID starting with 8cc8225fa592c6e6278ae73ec4a65853e7180992187efb1a54d1e03b3f9d21f5 not found: ID does not exist"
Nov 01 20:24:35.071400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.071372    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"]
Nov 01 20:24:35.074611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.074588    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aaea41-54c4d9455-5cd64"]
Nov 01 20:24:35.085847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.085821    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"]
Nov 01 20:24:35.089576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.089553    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaea41-db75c5f57-kqfg4"]
Nov 01 20:24:35.123193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.123167    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg9c\" (UniqueName: \"kubernetes.io/projected/bacf3d06-22d0-4637-bc28-991f58a055ad-kube-api-access-dzg9c\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:35.123193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.123193    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwzpv\" (UniqueName: \"kubernetes.io/projected/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-kube-api-access-lwzpv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:35.123330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.123204    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:35.123330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.123213    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacf3d06-22d0-4637-bc28-991f58a055ad-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:35.263269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.263233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:35.783690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.783629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:35.890177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:35.890081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:36.049324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.049287    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-63c56c" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" containerName="test-curl-7417-63c56c" containerID="cri-o://4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa" gracePeriod=2
Nov 01 20:24:36.051665 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:36.051620    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-63c56c\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:36.371981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.370038    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:36.437612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.437563    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:36.438332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.438297    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:36.439719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.439697    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:24:36.460575 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:36.460533    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-63c56c\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:36.461874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.461843    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:36.489666 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:36.488894    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-63c56c\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:36.718175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.718089    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" path="/var/lib/kubelet/pods/0a8f350c-b464-493d-8a6c-ef9bd7a9daa1/volumes"
Nov 01 20:24:36.718854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.718827    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" path="/var/lib/kubelet/pods/bacf3d06-22d0-4637-bc28-991f58a055ad/volumes"
Nov 01 20:24:36.725471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:36.725439    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:37.156917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:37.156815    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:37.303870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:37.303829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:37.943212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:37.943177    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:38.282000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:38.281975    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:38.402704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:38.402642    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr\" (UniqueName: \"kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr\") pod \"8e42d602-3197-4d66-b8b4-7e22b22dc637\" (UID: \"8e42d602-3197-4d66-b8b4-7e22b22dc637\") "
Nov 01 20:24:38.404833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:38.404799    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr" pod "8e42d602-3197-4d66-b8b4-7e22b22dc637" (UID: "8e42d602-3197-4d66-b8b4-7e22b22dc637"). InnerVolumeSpecName "kube-api-access-dccjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:38.504691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:38.504577    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dccjr\" (UniqueName: \"kubernetes.io/projected/8e42d602-3197-4d66-b8b4-7e22b22dc637-kube-api-access-dccjr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:38.539982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:38.539948    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:38.714194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:38.713614    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" path="/var/lib/kubelet/pods/8e42d602-3197-4d66-b8b4-7e22b22dc637/volumes"
Nov 01 20:24:39.092958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.092919    6479 generic.go:356] "Generic (PLEG): container finished" podID="8e42d602-3197-4d66-b8b4-7e22b22dc637" containerID="4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa" exitCode=137
Nov 01 20:24:39.093457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.092974    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-63c56c"
Nov 01 20:24:39.093457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.093014    6479 scope.go:122] "RemoveContainer" containerID="4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa"
Nov 01 20:24:39.105123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.105098    6479 scope.go:122] "RemoveContainer" containerID="4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa"
Nov 01 20:24:39.105430 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:39.105407    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa\": container with ID starting with 4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa not found: ID does not exist" containerID="4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa"
Nov 01 20:24:39.105500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.105438    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa"} err="failed to get container status \"4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa\": rpc error: code = NotFound desc = could not find container \"4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa\": container with ID starting with 4141eb046a2214b503ad7b7a0f1dbcfd236a190877c6c772d463c3765a551afa not found: ID does not exist"
Nov 01 20:24:39.171480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.171428    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:39.801026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:39.800984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:40.450797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:40.450765    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:41.048716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:41.048661    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:41.719363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:41.719315    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:42.154271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:42.154183    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"]
Nov 01 20:24:42.239822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:42.239790    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:42.579766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:42.579717    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:42.686497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:42.686458    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:43.248852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:43.248809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:43.418189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:43.418153    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:43.870926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:43.870892    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:44.546858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:44.546820    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:44.786711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:44.786675    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:45.121899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.121863    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"]
Nov 01 20:24:45.123015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.122990    6479 cpu_manager.go:414] "containerMap: removing container" podUID="c3953f2b-58ea-4e45-9308-9dedfba106c2" containerName="maas-api"
Nov 01 20:24:45.123105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123017    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3953f2b-58ea-4e45-9308-9dedfba106c2" containerName="maas-api"
Nov 01 20:24:45.123105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123050    6479 cpu_manager.go:414] "containerMap: removing container" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerName="payload-processing"
Nov 01 20:24:45.123105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123058    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerName="payload-processing"
Nov 01 20:24:45.123195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123106    6479 cpu_manager.go:414] "containerMap: removing container" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerName="payload-pre-processing"
Nov 01 20:24:45.123195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123116    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerName="payload-pre-processing"
Nov 01 20:24:45.123195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123150    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" containerName="test-curl-7417-63c56c"
Nov 01 20:24:45.123195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123157    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" containerName="test-curl-7417-63c56c"
Nov 01 20:24:45.123322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123308    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8f350c-b464-493d-8a6c-ef9bd7a9daa1" containerName="payload-pre-processing"
Nov 01 20:24:45.123366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123329    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e42d602-3197-4d66-b8b4-7e22b22dc637" containerName="test-curl-7417-63c56c"
Nov 01 20:24:45.123366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123344    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3953f2b-58ea-4e45-9308-9dedfba106c2" containerName="maas-api"
Nov 01 20:24:45.123366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.123363    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacf3d06-22d0-4637-bc28-991f58a055ad" containerName="payload-processing"
Nov 01 20:24:45.129202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.129181    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.136167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.136138    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-24b244-openshift-default-dockercfg-wq2h5\""
Nov 01 20:24:45.137853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.137826    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"]
Nov 01 20:24:45.153857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.153823    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:45.226328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226284    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226391    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226514    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226551    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226732    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226774    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226849    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.226953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.226937    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.227074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.227061    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2497q\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.244362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.244334    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:45.331191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.331144    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.331993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.331793    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.332343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.332322    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.333071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.333053    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.333751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.333730    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2497q\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.356636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.333000    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.356636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.356038    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.356636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.356488    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:45.357542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.357507    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.358296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.357810    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.373699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.358617    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.377215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.375031    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.377872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.377548    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.378022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.377816    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.378139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.377166    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.385633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.385603    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2497q\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.425025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.424990    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.427321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.427230    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.428169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.428146    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo\") pod \"e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") " pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.456672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.454980    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:45.638013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.637933    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"]
Nov 01 20:24:45.657290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.657252    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"]
Nov 01 20:24:45.698984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.698826    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:45.799184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.799149    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:45.825445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.825417    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"]
Nov 01 20:24:45.826007 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:24:45.825982    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf15910b_5a7e_4a09_9ea2_26ba8365cd67.slice/crio-494f9fbbf67351bbb80c627cbc2f89d30fb93603f0a167f8ae0729e2f8f45191 WatchSource:0}: Error finding container 494f9fbbf67351bbb80c627cbc2f89d30fb93603f0a167f8ae0729e2f8f45191: Status 404 returned error can't find the container with id 494f9fbbf67351bbb80c627cbc2f89d30fb93603f0a167f8ae0729e2f8f45191
Nov 01 20:24:45.828428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.828396    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:24:45.828508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.828460    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:24:45.828508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.828489    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:24:45.980719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:45.979942    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:46.237663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:46.235552    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" event={"ID":"bf15910b-5a7e-4a09-9ea2-26ba8365cd67","Type":"ContainerStarted","Data":"ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"}
Nov 01 20:24:46.237663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:46.235629    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" event={"ID":"bf15910b-5a7e-4a09-9ea2-26ba8365cd67","Type":"ContainerStarted","Data":"494f9fbbf67351bbb80c627cbc2f89d30fb93603f0a167f8ae0729e2f8f45191"}
Nov 01 20:24:46.259716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:46.259111    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podStartSLOduration=1.259092833 podStartE2EDuration="1.259092833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:46.255637885 +0000 UTC m=+1531.671270360" watchObservedRunningTime="2026-11-01 20:24:46.259092833 +0000 UTC m=+1531.674725311"
Nov 01 20:24:46.463137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:46.460640    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:46.534628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:46.534534    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:24:46.756693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:46.756657    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:47.462110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:47.462080    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:47.642469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:47.642430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:48.250461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.250429    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:48.251367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.251334    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:24:48.302920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.302886    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"]
Nov 01 20:24:48.358494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.358463    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:48.457770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.457720    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:48.561806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.561690    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:48.664287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.664230    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:48.784921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.784877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:48.863741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:48.863113    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:49.736161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.736124    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:49.938290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.938254    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:49.948679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.948637    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"]
Nov 01 20:24:49.955899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.955872    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:49.962300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.962267    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"]
Nov 01 20:24:49.967408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.967386    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-24b244\""
Nov 01 20:24:49.967690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.967673    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-24b244-dockercfg-scgzr\""
Nov 01 20:24:49.998879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:49.998818    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"]
Nov 01 20:24:50.009863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.009832    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"]
Nov 01 20:24:50.010014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.009947    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.035231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.035198    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.035601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.035569    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:50.035818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.035793    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnn7\" (UniqueName: \"kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7\") pod \"payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.036028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.036008    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dmj4\" (UniqueName: \"kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4\") pod \"payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.036233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.036215    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.139674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.137608    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.139674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.137747    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.139674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.137907    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnn7\" (UniqueName: \"kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7\") pod \"payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.139674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.137998    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmj4\" (UniqueName: \"kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4\") pod \"payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.154681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.141787    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.163754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.162790    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.174744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.170498    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmj4\" (UniqueName: \"kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4\") pod \"payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.213249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.213215    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnn7\" (UniqueName: \"kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7\") pod \"payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.316012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.315913    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:50.355892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.355854    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:50.638027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.637943    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:50.638722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.638686    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerName="payload-processing" containerID="cri-o://40780a4d5f6e40dd73a129a49797f408b1d76890acadfc812f24174db40a9c3a" gracePeriod=25
Nov 01 20:24:50.658280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.658236    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerName="payload-pre-processing" containerID="cri-o://8e830b254402312625d6b9a149f42f19a87f76e96f8b218612f2b5e7217cc0fe" gracePeriod=25
Nov 01 20:24:50.714714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.692509    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"]
Nov 01 20:24:50.714714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:50.697962    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"]
Nov 01 20:24:51.354631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.349170    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" event={"ID":"30e49bc0-8968-4d82-ba88-62b0199fa183","Type":"ContainerStarted","Data":"9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6"}
Nov 01 20:24:51.354631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.349288    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" event={"ID":"30e49bc0-8968-4d82-ba88-62b0199fa183","Type":"ContainerStarted","Data":"fcbb8bd2db56a1ef68ffaa3ba6c23ddd23f69773f8ebfb6d4aa97a029f42ad98"}
Nov 01 20:24:51.354631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.354071    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:24:51.366463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.366271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:51.371691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.367415    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" event={"ID":"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c","Type":"ContainerStarted","Data":"92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce"}
Nov 01 20:24:51.371691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.367474    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" event={"ID":"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c","Type":"ContainerStarted","Data":"33dcbae3e43795bf0021ac565005cd92c7d470887f1822a6ddbf3cc24577aa12"}
Nov 01 20:24:51.371691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.368350    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:24:51.424669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.424019    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" podStartSLOduration=2.42400197 podStartE2EDuration="2.42400197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:51.42289898 +0000 UTC m=+1536.838531456" watchObservedRunningTime="2026-11-01 20:24:51.42400197 +0000 UTC m=+1536.839634480"
Nov 01 20:24:51.441445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.441387    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" podStartSLOduration=2.441372312 podStartE2EDuration="2.441372312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:51.437697103 +0000 UTC m=+1536.853329576" watchObservedRunningTime="2026-11-01 20:24:51.441372312 +0000 UTC m=+1536.857004784"
Nov 01 20:24:51.955130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:51.955095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:52.634489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:52.634454    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:53.240800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:53.240767    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:53.308675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:53.307493    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerName="payload-processing" containerID="cri-o://bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f" gracePeriod=25
Nov 01 20:24:53.642746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:53.642627    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:53.643158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:53.642810    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:53.907230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:53.907115    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:54.541371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:54.541336    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:55.160127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:55.160093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:55.289352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:55.289092    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:55.745550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:55.745516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:56.436098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:56.436057    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:56.695136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:56.695096    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:24:56.999928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:56.999844    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"]
Nov 01 20:24:57.004598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.004576    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.013358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.013331    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-24b244-serving-cert\""
Nov 01 20:24:57.025777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.025019    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"]
Nov 01 20:24:57.050559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.050520    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:57.105040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.105002    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.105224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.105204    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.105359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.105343    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbvn\" (UniqueName: \"kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.105414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.105373    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.161910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.161842    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:57.205208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.205140    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" containerID="cri-o://32e70d55a99dd1d70ceb6b18699553c7983a6128cacbbfc3fea00421b4b37566" gracePeriod=45
Nov 01 20:24:57.208240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.208178    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.208403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.208390    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbvn\" (UniqueName: \"kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.208476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.208449    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.208569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.208550    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.216137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.216107    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.228705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.228675    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.240713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.231241    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbvn\" (UniqueName: \"kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.240713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.234376    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs\") pod \"maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.254740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.253039    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": dial tcp 10.134.0.36:8000: connect: connection refused"
Nov 01 20:24:57.254740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.254405    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:57.332448 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.332418    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:57.443982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.443432    6479 generic.go:356] "Generic (PLEG): container finished" podID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerID="32e70d55a99dd1d70ceb6b18699553c7983a6128cacbbfc3fea00421b4b37566" exitCode=0
Nov 01 20:24:57.443982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.443574    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" event={"ID":"8f79ac23-e682-4678-827f-3fdf4b5523c7","Type":"ContainerDied","Data":"32e70d55a99dd1d70ceb6b18699553c7983a6128cacbbfc3fea00421b4b37566"}
Nov 01 20:24:57.606596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.606565    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:24:57.636760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.636725    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"]
Nov 01 20:24:57.717120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717090    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache\") pod \"8f79ac23-e682-4678-827f-3fdf4b5523c7\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") "
Nov 01 20:24:57.717496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717138    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir\") pod \"8f79ac23-e682-4678-827f-3fdf4b5523c7\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") "
Nov 01 20:24:57.717496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717180    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home\") pod \"8f79ac23-e682-4678-827f-3fdf4b5523c7\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") "
Nov 01 20:24:57.717496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717201    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm\") pod \"8f79ac23-e682-4678-827f-3fdf4b5523c7\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") "
Nov 01 20:24:57.717496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717224    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs\") pod \"8f79ac23-e682-4678-827f-3fdf4b5523c7\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") "
Nov 01 20:24:57.717496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717406    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl\" (UniqueName: \"kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl\") pod \"8f79ac23-e682-4678-827f-3fdf4b5523c7\" (UID: \"8f79ac23-e682-4678-827f-3fdf4b5523c7\") "
Nov 01 20:24:57.717496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717476    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home" pod "8f79ac23-e682-4678-827f-3fdf4b5523c7" (UID: "8f79ac23-e682-4678-827f-3fdf4b5523c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.717901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.717833    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache" pod "8f79ac23-e682-4678-827f-3fdf4b5523c7" (UID: "8f79ac23-e682-4678-827f-3fdf4b5523c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.718068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.718034    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.718068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.718062    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.718197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.718152    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir" pod "8f79ac23-e682-4678-827f-3fdf4b5523c7" (UID: "8f79ac23-e682-4678-827f-3fdf4b5523c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.719543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.719520    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm" pod "8f79ac23-e682-4678-827f-3fdf4b5523c7" (UID: "8f79ac23-e682-4678-827f-3fdf4b5523c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.719826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.719806    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs" pod "8f79ac23-e682-4678-827f-3fdf4b5523c7" (UID: "8f79ac23-e682-4678-827f-3fdf4b5523c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:57.719920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.719897    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl" pod "8f79ac23-e682-4678-827f-3fdf4b5523c7" (UID: "8f79ac23-e682-4678-827f-3fdf4b5523c7"). InnerVolumeSpecName "kube-api-access-khpwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:57.820683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.820580    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khpwl\" (UniqueName: \"kubernetes.io/projected/8f79ac23-e682-4678-827f-3fdf4b5523c7-kube-api-access-khpwl\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.820683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.820624    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.820683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.820640    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f79ac23-e682-4678-827f-3fdf4b5523c7-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.820683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.820670    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f79ac23-e682-4678-827f-3fdf4b5523c7-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.924021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:57.923930    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:58.454493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.454401    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"
Nov 01 20:24:58.454493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.454405    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj" event={"ID":"8f79ac23-e682-4678-827f-3fdf4b5523c7","Type":"ContainerDied","Data":"cfb2e26b998b5dab91305d096bda922c319039f250537e24590aaefade828b07"}
Nov 01 20:24:58.455032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.454548    6479 scope.go:122] "RemoveContainer" containerID="32e70d55a99dd1d70ceb6b18699553c7983a6128cacbbfc3fea00421b4b37566"
Nov 01 20:24:58.456510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.456482    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn" event={"ID":"4a4eae91-2776-4013-b4b1-48b3356b1b61","Type":"ContainerStarted","Data":"59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba"}
Nov 01 20:24:58.456660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.456532    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn" event={"ID":"4a4eae91-2776-4013-b4b1-48b3356b1b61","Type":"ContainerStarted","Data":"3df09012963a702af2e2054d49f9690c05eb919e26fd01520b3548abeed10992"}
Nov 01 20:24:58.456660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.456599    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:24:58.476924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.476881    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn" podStartSLOduration=1.975084527 podStartE2EDuration="2.476867662s" totalImagesPullingTime="501.783135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:58.475403204 +0000 UTC m=+1543.891035678" watchObservedRunningTime="2026-11-01 20:24:58.476867662 +0000 UTC m=+1543.892500134"
Nov 01 20:24:58.493970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.493940    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"]
Nov 01 20:24:58.502264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.497805    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/simulator-w2-e27ae7-kserve-75495dd7d8-dn9tj"]
Nov 01 20:24:58.540625 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.540551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:58.709026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:58.708953    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" path="/var/lib/kubelet/pods/8f79ac23-e682-4678-827f-3fdf4b5523c7/volumes"
Nov 01 20:24:59.146438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:59.146348    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:24:59.171465 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:24:59.171429    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:24:59.782989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:24:59.782953    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:00.463282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:00.463252    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:01.050118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:01.050081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:01.485039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:01.484999    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:01.647485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:01.647453    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:02.242603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:02.242571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:02.381014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:02.380181    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:25:02.832991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:02.832944    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:03.404913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:03.404886    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:25:03.442529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:03.442497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:03.643148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:03.643114    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:25:03.643327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:03.643115    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:25:04.034919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:04.034881    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:04.417255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:04.417170    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:04.471741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:04.471714    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:25:04.519802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:04.519747    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:04.620228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:04.620184    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:04.719966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:04.719936    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:05.436211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:05.436176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:05.539562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:05.539529    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:05.647928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:05.647881    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:06.352269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:06.352231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:06.695103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:06.695069    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:25:07.410761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:07.410725    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:07.905654 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:07.905605    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:25:08.408537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:08.408502    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:08.522800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:08.522752    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:09.384809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:09.384767    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:09.628806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:09.628756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:10.067540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:10.067497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:10.458119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:10.452485    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:10.787683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:10.785710    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:11.076260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.076160    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:11.239673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.238146    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" podUID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": context deadline exceeded"
Nov 01 20:25:11.279212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.242797    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Nov 01 20:25:11.279212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.246272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:11.279212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.258128    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:11.301136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.301087    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:11.319671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.302368    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:11.921412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:11.921344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:12.141886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.131472    6479 generic.go:356] "Generic (PLEG): container finished" podID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerID="40780a4d5f6e40dd73a129a49797f408b1d76890acadfc812f24174db40a9c3a" exitCode=0
Nov 01 20:25:12.141886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.131575    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" event={"ID":"cf6e18e8-ada2-4755-be98-6871e43c8d43","Type":"ContainerDied","Data":"40780a4d5f6e40dd73a129a49797f408b1d76890acadfc812f24174db40a9c3a"}
Nov 01 20:25:12.141886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.131626    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29" event={"ID":"cf6e18e8-ada2-4755-be98-6871e43c8d43","Type":"ContainerDied","Data":"89cc98bbd6e97b0afdacaa54e2a57aaa57c08603dac40c78f74bd5b7988c1d4b"}
Nov 01 20:25:12.141886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.131666    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89cc98bbd6e97b0afdacaa54e2a57aaa57c08603dac40c78f74bd5b7988c1d4b"
Nov 01 20:25:12.141886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.134011    6479 generic.go:356] "Generic (PLEG): container finished" podID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerID="8e830b254402312625d6b9a149f42f19a87f76e96f8b218612f2b5e7217cc0fe" exitCode=0
Nov 01 20:25:12.141886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.134113    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" event={"ID":"e7110dd3-c906-47ae-b79a-98939a9e0d52","Type":"ContainerDied","Data":"8e830b254402312625d6b9a149f42f19a87f76e96f8b218612f2b5e7217cc0fe"}
Nov 01 20:25:12.230116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.226556    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:25:12.231407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.231380    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:25:12.328314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.328280    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume\") pod \"cf6e18e8-ada2-4755-be98-6871e43c8d43\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") "
Nov 01 20:25:12.328482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.328459    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume\") pod \"e7110dd3-c906-47ae-b79a-98939a9e0d52\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") "
Nov 01 20:25:12.328522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.328496    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8\" (UniqueName: \"kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8\") pod \"cf6e18e8-ada2-4755-be98-6871e43c8d43\" (UID: \"cf6e18e8-ada2-4755-be98-6871e43c8d43\") "
Nov 01 20:25:12.328609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.328598    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24\" (UniqueName: \"kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24\") pod \"e7110dd3-c906-47ae-b79a-98939a9e0d52\" (UID: \"e7110dd3-c906-47ae-b79a-98939a9e0d52\") "
Nov 01 20:25:12.329937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.329909    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume" pod "cf6e18e8-ada2-4755-be98-6871e43c8d43" (UID: "cf6e18e8-ada2-4755-be98-6871e43c8d43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:12.330235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.330217    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume" pod "e7110dd3-c906-47ae-b79a-98939a9e0d52" (UID: "e7110dd3-c906-47ae-b79a-98939a9e0d52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:12.332229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.332202    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24" pod "e7110dd3-c906-47ae-b79a-98939a9e0d52" (UID: "e7110dd3-c906-47ae-b79a-98939a9e0d52"). InnerVolumeSpecName "kube-api-access-zzl24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:12.332323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.332209    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8" pod "cf6e18e8-ada2-4755-be98-6871e43c8d43" (UID: "cf6e18e8-ada2-4755-be98-6871e43c8d43"). InnerVolumeSpecName "kube-api-access-j9lz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:12.430391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.430353    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7110dd3-c906-47ae-b79a-98939a9e0d52-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:12.430391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.430386    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9lz8\" (UniqueName: \"kubernetes.io/projected/cf6e18e8-ada2-4755-be98-6871e43c8d43-kube-api-access-j9lz8\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:12.430391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.430396    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzl24\" (UniqueName: \"kubernetes.io/projected/e7110dd3-c906-47ae-b79a-98939a9e0d52-kube-api-access-zzl24\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:12.430639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.430406    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6e18e8-ada2-4755-be98-6871e43c8d43-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:12.884410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:12.884374    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:13.142999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.142965    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"
Nov 01 20:25:13.142999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.142989    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l" event={"ID":"e7110dd3-c906-47ae-b79a-98939a9e0d52","Type":"ContainerDied","Data":"a91a416135fb29d766a6d59c5268e23bbb07ab6d2de556b44a4566572d217c2a"}
Nov 01 20:25:13.143497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.143075    6479 scope.go:122] "RemoveContainer" containerID="8e830b254402312625d6b9a149f42f19a87f76e96f8b218612f2b5e7217cc0fe"
Nov 01 20:25:13.143497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.143286    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"
Nov 01 20:25:13.188435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.188404    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"]
Nov 01 20:25:13.191839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.191813    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-kvt29"]
Nov 01 20:25:13.195425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.195399    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"]
Nov 01 20:25:13.199451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.199430    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-rt47l"]
Nov 01 20:25:13.450797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.450752    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:13.609478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.609402    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:25:13.678668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.668276    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:13.703406 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.703374    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume\") pod \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") "
Nov 01 20:25:13.703601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.703434    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj\" (UniqueName: \"kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj\") pod \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\" (UID: \"1d0841ea-4959-469d-a6c7-dfff4d489cb2\") "
Nov 01 20:25:13.704100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.704064    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume" pod "1d0841ea-4959-469d-a6c7-dfff4d489cb2" (UID: "1d0841ea-4959-469d-a6c7-dfff4d489cb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:13.706838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.706811    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj" pod "1d0841ea-4959-469d-a6c7-dfff4d489cb2" (UID: "1d0841ea-4959-469d-a6c7-dfff4d489cb2"). InnerVolumeSpecName "kube-api-access-4dhwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:13.818682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.816794    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:13.826680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.824595    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0841ea-4959-469d-a6c7-dfff4d489cb2-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:13.826680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:13.824633    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhwj\" (UniqueName: \"kubernetes.io/projected/1d0841ea-4959-469d-a6c7-dfff4d489cb2-kube-api-access-4dhwj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:14.191630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.189527    6479 generic.go:356] "Generic (PLEG): container finished" podID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerID="bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f" exitCode=0
Nov 01 20:25:14.191630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.189707    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" event={"ID":"1d0841ea-4959-469d-a6c7-dfff4d489cb2","Type":"ContainerDied","Data":"bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f"}
Nov 01 20:25:14.191630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.189792    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6" event={"ID":"1d0841ea-4959-469d-a6c7-dfff4d489cb2","Type":"ContainerDied","Data":"f17fe315d8dc4948631f86bbc9e25168d18b68153fa9170f79406daa9e90c303"}
Nov 01 20:25:14.191630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.189813    6479 scope.go:122] "RemoveContainer" containerID="bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f"
Nov 01 20:25:14.191630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.189940    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"
Nov 01 20:25:14.205984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.205955    6479 scope.go:122] "RemoveContainer" containerID="bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f"
Nov 01 20:25:14.206518 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:25:14.206493    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f\": container with ID starting with bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f not found: ID does not exist" containerID="bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f"
Nov 01 20:25:14.206903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.206881    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f"} err="failed to get container status \"bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f\": rpc error: code = NotFound desc = could not find container \"bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f\": container with ID starting with bdae155195f2219bd25ccb2580b71433d57c1b66a6a2335197333a0126f0c06f not found: ID does not exist"
Nov 01 20:25:14.219203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.219180    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"]
Nov 01 20:25:14.228946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.228922    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e27ae7-7dc769584f-rrjx6"]
Nov 01 20:25:14.608149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.608107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:14.736698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.736003    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" path="/var/lib/kubelet/pods/1d0841ea-4959-469d-a6c7-dfff4d489cb2/volumes"
Nov 01 20:25:14.736896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.736756    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" path="/var/lib/kubelet/pods/cf6e18e8-ada2-4755-be98-6871e43c8d43/volumes"
Nov 01 20:25:14.737354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.737334    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" path="/var/lib/kubelet/pods/e7110dd3-c906-47ae-b79a-98939a9e0d52/volumes"
Nov 01 20:25:14.995561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:14.995484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:15.267721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:15.248567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:15.643583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:15.643500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:16.245704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:16.245639    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:16.345960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:16.345877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:16.451678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:16.447037    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:17.207070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:17.207042    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:17.313151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:17.313073    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:17.922541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:17.922492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:18.026570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.026514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:18.142669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.140628    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:18.151375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.151332    6479 scope.go:122] "RemoveContainer" containerID="c0444cbb3b5336701d8d786ef514e1cb7ff2ee9b6311cdc2dc56c5ede99a9bb8"
Nov 01 20:25:18.369883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.369845    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:18.376422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.376380    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"]
Nov 01 20:25:18.377344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377321    6479 cpu_manager.go:414] "containerMap: removing container" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerName="payload-processing"
Nov 01 20:25:18.377344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377340    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerName="payload-processing"
Nov 01 20:25:18.377521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377375    6479 cpu_manager.go:414] "containerMap: removing container" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerName="payload-processing"
Nov 01 20:25:18.377521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377384    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerName="payload-processing"
Nov 01 20:25:18.377521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377403    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerName="payload-pre-processing"
Nov 01 20:25:18.377521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377412    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerName="payload-pre-processing"
Nov 01 20:25:18.377521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377434    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main"
Nov 01 20:25:18.377521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377442    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main"
Nov 01 20:25:18.377862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377596    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0841ea-4959-469d-a6c7-dfff4d489cb2" containerName="payload-processing"
Nov 01 20:25:18.377862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377613    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6e18e8-ada2-4755-be98-6871e43c8d43" containerName="payload-processing"
Nov 01 20:25:18.377862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377624    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7110dd3-c906-47ae-b79a-98939a9e0d52" containerName="payload-pre-processing"
Nov 01 20:25:18.377862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.377635    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f79ac23-e682-4678-827f-3fdf4b5523c7" containerName="main"
Nov 01 20:25:18.383853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.383829    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.388893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.388867    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"]
Nov 01 20:25:18.391315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.391289    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2babaef6\""
Nov 01 20:25:18.392416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.392393    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2babaef6-dockercfg-pv9ff\""
Nov 01 20:25:18.402086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.401663    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"]
Nov 01 20:25:18.408903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.407855    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.415708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.415681    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"]
Nov 01 20:25:18.449984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.449945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:18.556782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.546330    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.556782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.546488    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqgq\" (UniqueName: \"kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.556782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.546569    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.556782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.546627    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qbp\" (UniqueName: \"kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.650121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.650076    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqgq\" (UniqueName: \"kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.650460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.650435    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.650662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.650633    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qbp\" (UniqueName: \"kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.650887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.650872    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.652019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.651995    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.653342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.653321    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.673528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.673473    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:18.717678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.710687    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qbp\" (UniqueName: \"kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp\") pod \"payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") " pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.729753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.725789    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:18.733624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.733492    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqgq\" (UniqueName: \"kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq\") pod \"payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.733830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.733806    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:18.786920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:18.786385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:19.211703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.197579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:19.291862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.291784    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:25:19.291862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.291851    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:19.502676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.490699    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"]
Nov 01 20:25:19.529938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.529854    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"]
Nov 01 20:25:19.536970 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:25:19.536878    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d12e3c4_b878_4c26_8e0f_7a517b00de25.slice/crio-0032b4e4c29a302816ccb225be6ff474abbb3fb67f82b90260dfceba85ff060e WatchSource:0}: Error finding container 0032b4e4c29a302816ccb225be6ff474abbb3fb67f82b90260dfceba85ff060e: Status 404 returned error can't find the container with id 0032b4e4c29a302816ccb225be6ff474abbb3fb67f82b90260dfceba85ff060e
Nov 01 20:25:19.743530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.738273    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk" event={"ID":"797d99ae-3ddb-4af7-a167-cbfd44181dde","Type":"ContainerStarted","Data":"e4acd13d8166ff81538c20f5c18040ce33e4f25ab9267ae1bfd038e0bededbec"}
Nov 01 20:25:19.743530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:19.740485    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v" event={"ID":"3d12e3c4-b878-4c26-8e0f-7a517b00de25","Type":"ContainerStarted","Data":"0032b4e4c29a302816ccb225be6ff474abbb3fb67f82b90260dfceba85ff060e"}
Nov 01 20:25:20.056926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.056844    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:20.629125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.629084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:20.751361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.751324    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v" event={"ID":"3d12e3c4-b878-4c26-8e0f-7a517b00de25","Type":"ContainerStarted","Data":"a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711"}
Nov 01 20:25:20.751534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.751410    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:20.753332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.753294    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk" event={"ID":"797d99ae-3ddb-4af7-a167-cbfd44181dde","Type":"ContainerStarted","Data":"8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d"}
Nov 01 20:25:20.753451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.753434    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:20.770408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.770355    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v" podStartSLOduration=2.770326491 podStartE2EDuration="2.770326491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:20.768987383 +0000 UTC m=+1566.184619856" watchObservedRunningTime="2026-11-01 20:25:20.770326491 +0000 UTC m=+1566.185958965"
Nov 01 20:25:20.785382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.785337    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk" podStartSLOduration=2.785326187 podStartE2EDuration="2.785326187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:20.784130867 +0000 UTC m=+1566.199763340" watchObservedRunningTime="2026-11-01 20:25:20.785326187 +0000 UTC m=+1566.200958660"
Nov 01 20:25:20.812342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:20.812299    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:21.049389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.049355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:21.159798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.159761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:21.255785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.255751    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:21.348169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.348068    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"]
Nov 01 20:25:21.348351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.348322    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" containerID="cri-o://c95132d6627bffa0dce49f650bd6e35a730788988408ebd2e02d45d7548834d3" gracePeriod=30
Nov 01 20:25:21.368674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.356829    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"]
Nov 01 20:25:21.368674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.357058    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy" containerID="cri-o://837c27fe81505faf2276e492d9c91897b7f068e3dd3e0cbb21d4879740b74f06" gracePeriod=30
Nov 01 20:25:21.368674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.361368    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"]
Nov 01 20:25:21.368674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.361593    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" containerID="cri-o://f597d0ff225d22937cc1b5fb36d5d89e555c0f2b3dd63a97b897c770c88566f5" gracePeriod=30
Nov 01 20:25:21.386200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.386175    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:21.440305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.440267    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:25:21.440489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.440320    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:25:21.762735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.762696    6479 generic.go:356] "Generic (PLEG): container finished" podID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerID="ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b" exitCode=0
Nov 01 20:25:21.763212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.762759    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" event={"ID":"bf15910b-5a7e-4a09-9ea2-26ba8365cd67","Type":"ContainerDied","Data":"ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"}
Nov 01 20:25:21.763212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.763205    6479 scope.go:122] "RemoveContainer" containerID="ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"
Nov 01 20:25:21.764844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.764812    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:25:21.764987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.764884    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:25:21.764987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.764915    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:25:21.866742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:21.866624    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:22.746750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:22.746712    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:22.781778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:22.781743    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" event={"ID":"bf15910b-5a7e-4a09-9ea2-26ba8365cd67","Type":"ContainerStarted","Data":"4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3"}
Nov 01 20:25:22.877734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:22.877618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:23.460672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:23.459070    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:25:23.460672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:23.459205    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 01 20:25:23.460672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:23.459247    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 01 20:25:23.648903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:23.648869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:23.756380 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:23.756290    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:24.257883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:24.250965    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:24.351790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:24.351755    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:24.431163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:24.431128    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:24.457333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:24.457298    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 01 20:25:24.457630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:24.457593    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 01 20:25:25.012288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:25.012258    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:25.483671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:25.483123    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 01 20:25:25.483671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:25.483190    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 01 20:25:25.830128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:25.830034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:25.921420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:25.921378    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:26.050562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.050527    6479 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:25:26.050773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.050722    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:25:26.476859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.476819    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 01 20:25:26.477167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.477134    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 01 20:25:26.585849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.585780    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:26.814205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.812277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:26.831827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.831782    6479 generic.go:356] "Generic (PLEG): container finished" podID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerID="f597d0ff225d22937cc1b5fb36d5d89e555c0f2b3dd63a97b897c770c88566f5" exitCode=0
Nov 01 20:25:26.831982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.831966    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" event={"ID":"34e99777-54a2-4ff5-875f-8d88ef07caeb","Type":"ContainerDied","Data":"f597d0ff225d22937cc1b5fb36d5d89e555c0f2b3dd63a97b897c770c88566f5"}
Nov 01 20:25:26.832347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.832325    6479 scope.go:122] "RemoveContainer" containerID="4050a378770af736146167332d30d55928f8a1af18022011c55cf7be6de447e3"
Nov 01 20:25:26.840545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.840372    6479 generic.go:356] "Generic (PLEG): container finished" podID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerID="c95132d6627bffa0dce49f650bd6e35a730788988408ebd2e02d45d7548834d3" exitCode=0
Nov 01 20:25:26.840545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.840458    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" event={"ID":"b30ec562-4f03-46d5-a99a-3e34973a3e8f","Type":"ContainerDied","Data":"c95132d6627bffa0dce49f650bd6e35a730788988408ebd2e02d45d7548834d3"}
Nov 01 20:25:26.843419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.843382    6479 generic.go:356] "Generic (PLEG): container finished" podID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerID="837c27fe81505faf2276e492d9c91897b7f068e3dd3e0cbb21d4879740b74f06" exitCode=0
Nov 01 20:25:26.843553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.843452    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" event={"ID":"f2a7451a-fc38-4af7-bace-2d2c11463a48","Type":"ContainerDied","Data":"837c27fe81505faf2276e492d9c91897b7f068e3dd3e0cbb21d4879740b74f06"}
Nov 01 20:25:26.899672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:26.897113    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:27.014453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.014419    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:25:27.024046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.024025    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:25:27.105033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.104998    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.105258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.105242    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.105372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.105360    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.105495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.105482    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.105617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.105605    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.105759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.105746    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.105894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.105882    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.106173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.106160    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.106354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.106342    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs\") pod \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\" (UID: \"b30ec562-4f03-46d5-a99a-3e34973a3e8f\") "
Nov 01 20:25:27.109167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.108415    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.109167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.108635    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.109167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.108843    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.109426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.109180    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.112409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.112379    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "kube-api-access-rlqkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.112707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.112674    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:27.121344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.120951    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:27.121344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.121293    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.122527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.122506    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token" pod "b30ec562-4f03-46d5-a99a-3e34973a3e8f" (UID: "b30ec562-4f03-46d5-a99a-3e34973a3e8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.207548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207517    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.207750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207561    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.207750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207664    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.207750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207723    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.207940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207785    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.207940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207822    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.207940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207867    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.208098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.207943    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.208098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208057    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket\") pod \"f2a7451a-fc38-4af7-bace-2d2c11463a48\" (UID: \"f2a7451a-fc38-4af7-bace-2d2c11463a48\") "
Nov 01 20:25:27.208585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208562    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208588    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208602    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208617    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208630    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208674    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqkp\" (UniqueName: \"kubernetes.io/projected/b30ec562-4f03-46d5-a99a-3e34973a3e8f-kube-api-access-rlqkp\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208689    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.208715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208704    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ec562-4f03-46d5-a99a-3e34973a3e8f-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.209138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.208718    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ec562-4f03-46d5-a99a-3e34973a3e8f-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.209266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.209233    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.209479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.209461    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.212745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.211285    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.212745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.211902    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:27.212745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.212714    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.213564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.213271    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:25:27.214928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.214900    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "kube-api-access-sqln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.218028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.218007    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.224043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.224005    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:27.224043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.224021    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy" pod "f2a7451a-fc38-4af7-bace-2d2c11463a48" (UID: "f2a7451a-fc38-4af7-bace-2d2c11463a48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.310317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.310240    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.310553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.310539    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.310969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.310951    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.311194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.310846    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.311279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.311189    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.311376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.311364    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.311525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.311509    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.312053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.311893    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.312361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.311828    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.312361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.312251    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:27.312553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.312393    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:27.312666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.312634    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.312845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.312803    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.313017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.312999    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.313174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.313155    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy\") pod \"34e99777-54a2-4ff5-875f-8d88ef07caeb\" (UID: \"34e99777-54a2-4ff5-875f-8d88ef07caeb\") "
Nov 01 20:25:27.313988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.313971    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.314128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314114    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.314271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314256    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.314417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314402    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.314562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314548    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e99777-54a2-4ff5-875f-8d88ef07caeb-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.314720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314704    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.314968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314954    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.315119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.315105    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.315253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.315239    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.315494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314013    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "kube-api-access-jcg4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.315494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.314907    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.316698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.316679    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqln7\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-kube-api-access-sqln7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.316826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.316812    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.316918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.316906    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.317007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.316995    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a7451a-fc38-4af7-bace-2d2c11463a48-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.317089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.317077    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a7451a-fc38-4af7-bace-2d2c11463a48-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.317167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.317153    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.317231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.316908    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy" pod "34e99777-54a2-4ff5-875f-8d88ef07caeb" (UID: "34e99777-54a2-4ff5-875f-8d88ef07caeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.372698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.362052    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:27.434257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.431999    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.434257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.432050    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e99777-54a2-4ff5-875f-8d88ef07caeb-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.434257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.432078    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcg4g\" (UniqueName: \"kubernetes.io/projected/34e99777-54a2-4ff5-875f-8d88ef07caeb-kube-api-access-jcg4g\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.461478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.461029    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 01 20:25:27.461478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.461090    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 01 20:25:27.852117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.852076    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"
Nov 01 20:25:27.852117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.852113    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm" event={"ID":"34e99777-54a2-4ff5-875f-8d88ef07caeb","Type":"ContainerDied","Data":"d8d7fb324fe78b5984c9c72216f69498fbfff0ba90ab9565579d209ea74d3768"}
Nov 01 20:25:27.852834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.852169    6479 scope.go:122] "RemoveContainer" containerID="f597d0ff225d22937cc1b5fb36d5d89e555c0f2b3dd63a97b897c770c88566f5"
Nov 01 20:25:27.853973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.853929    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w" event={"ID":"b30ec562-4f03-46d5-a99a-3e34973a3e8f","Type":"ContainerDied","Data":"0744756e7f4ce2df44e5759af0f26a965ee26f69208af9f1fc26739bcce4c75d"}
Nov 01 20:25:27.854115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.853951    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"
Nov 01 20:25:27.884910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.882211    6479 scope.go:122] "RemoveContainer" containerID="c95132d6627bffa0dce49f650bd6e35a730788988408ebd2e02d45d7548834d3"
Nov 01 20:25:27.884910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.884059    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n" event={"ID":"f2a7451a-fc38-4af7-bace-2d2c11463a48","Type":"ContainerDied","Data":"7ebb8b507dc04a28fa6e8ec60bdc1715588b7b1be0c45cebb405b0dc9e65ac79"}
Nov 01 20:25:27.884910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.884420    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"
Nov 01 20:25:27.970411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.961692    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"]
Nov 01 20:25:27.973027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.972998    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-72kcm"]
Nov 01 20:25:27.983678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.983608    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"]
Nov 01 20:25:27.989209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.988821    6479 scope.go:122] "RemoveContainer" containerID="837c27fe81505faf2276e492d9c91897b7f068e3dd3e0cbb21d4879740b74f06"
Nov 01 20:25:27.990452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:27.990433    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-9h52n"]
Nov 01 20:25:28.003037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.003014    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"]
Nov 01 20:25:28.006119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.006094    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-l747w"]
Nov 01 20:25:28.066820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.066785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:28.460689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.460642    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:25:28.461160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.461140    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:25:28.461837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.461810    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:25:28.711727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.711606    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" path="/var/lib/kubelet/pods/34e99777-54a2-4ff5-875f-8d88ef07caeb/volumes"
Nov 01 20:25:28.712163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.712133    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" path="/var/lib/kubelet/pods/b30ec562-4f03-46d5-a99a-3e34973a3e8f/volumes"
Nov 01 20:25:28.712630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.712614    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" path="/var/lib/kubelet/pods/f2a7451a-fc38-4af7-bace-2d2c11463a48/volumes"
Nov 01 20:25:28.739984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:28.739951    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:29.171757 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:25:29.171668    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:25:29.328211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:29.328170    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:29.951134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:29.951104    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:30.606573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:30.606543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:31.260353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:31.260315    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:31.766817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:31.766622    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:25:31.766817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:31.766741    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:25:32.042079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:32.041993    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:32.650776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:32.650742    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:33.234322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:33.234290    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:33.854883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:33.854835    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:34.463528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:34.463489    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:35.058126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:35.058086    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:35.893017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:35.892974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:36.109781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:36.109741    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:25:36.774873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:36.774832    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:38.217697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:38.217620    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:38.805672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:38.792272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:39.348124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:39.348080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:39.842244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:39.842203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:39.939899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:39.939154    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:40.097322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.097233    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:40.364849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.364764    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:40.592086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592054    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-8514c7"]
Nov 01 20:25:40.592781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592763    6479 cpu_manager.go:414] "containerMap: removing container" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy"
Nov 01 20:25:40.592842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592783    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy"
Nov 01 20:25:40.592842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592797    6479 cpu_manager.go:414] "containerMap: removing container" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy"
Nov 01 20:25:40.592842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592804    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy"
Nov 01 20:25:40.592842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592840    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy"
Nov 01 20:25:40.592985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592846    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy"
Nov 01 20:25:40.592985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592946    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy"
Nov 01 20:25:40.592985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592955    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30ec562-4f03-46d5-a99a-3e34973a3e8f" containerName="istio-proxy"
Nov 01 20:25:40.592985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592965    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy"
Nov 01 20:25:40.592985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.592974    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a7451a-fc38-4af7-bace-2d2c11463a48" containerName="istio-proxy"
Nov 01 20:25:40.596524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.596504    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:40.606844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.606817    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-8514c7"]
Nov 01 20:25:40.609450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.609417    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\""
Nov 01 20:25:40.658364 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.658267    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dgh\" (UniqueName: \"kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh\") pod \"test-curl-7417-8514c7\" (UID: \"bf0600fc-7107-4aca-90cf-fb44da81d4bc\") " pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:40.714067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.714034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:40.760229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.760194    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dgh\" (UniqueName: \"kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh\") pod \"test-curl-7417-8514c7\" (UID: \"bf0600fc-7107-4aca-90cf-fb44da81d4bc\") " pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:40.815678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.815623    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dgh\" (UniqueName: \"kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh\") pod \"test-curl-7417-8514c7\" (UID: \"bf0600fc-7107-4aca-90cf-fb44da81d4bc\") " pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:40.911325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.911236    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:40.933554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:40.933522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:41.029377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.029337    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:41.272672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.269297    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-8514c7"]
Nov 01 20:25:41.324671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.322670    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:41.544892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.544857    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"]
Nov 01 20:25:41.546525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.546460    6479 cpu_manager.go:414] "containerMap: removing container" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy"
Nov 01 20:25:41.546525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.546484    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e99777-54a2-4ff5-875f-8d88ef07caeb" containerName="istio-proxy"
Nov 01 20:25:41.555487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.552391    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"]
Nov 01 20:25:41.555487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.552503    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.562431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.559974    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-24b244\"/\"agc-cred-9860b9c33675\""
Nov 01 20:25:41.562431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.560080    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-24b244\"/\"routing-overlay\""
Nov 01 20:25:41.562431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.560138    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-24b244\"/\"openshift-service-ca.crt\""
Nov 01 20:25:41.562431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.560193    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-24b244\"/\"payload-processing-external-model-plugins-e2e-extmod-a-24b244\""
Nov 01 20:25:41.562431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.561260    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-24b244\"/\"kube-root-ca.crt\""
Nov 01 20:25:41.562431 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.561344    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-24b244\"/\"payload-processing-external-model-e2e--f3ceaef3-dockercfg-slghj\""
Nov 01 20:25:41.742672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.741530    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.742672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.741679    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.742672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.741804    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvkc\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.742672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.741934    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.846585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.846477    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.846799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.846662    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.846885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.846797    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.846950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.846933    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvkc\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.848324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.848270    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.850016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.849954    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.852933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.852886    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.860845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.860789    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvkc\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc\") pod \"payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") " pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:41.897371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:41.897330    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:42.066284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.066239    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-8514c7" event={"ID":"bf0600fc-7107-4aca-90cf-fb44da81d4bc","Type":"ContainerStarted","Data":"4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b"}
Nov 01 20:25:42.066779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.066753    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-8514c7" event={"ID":"bf0600fc-7107-4aca-90cf-fb44da81d4bc","Type":"ContainerStarted","Data":"9406532dd021c3273d22758833fc4bc6d872331c23990fd5b0ec5e8458262c59"}
Nov 01 20:25:42.083450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.083372    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-8514c7" podStartSLOduration=2.083357238 podStartE2EDuration="2.083357238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:42.082898895 +0000 UTC m=+1587.498531367" watchObservedRunningTime="2026-11-01 20:25:42.083357238 +0000 UTC m=+1587.498989692"
Nov 01 20:25:42.150464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.150377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:42.196374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.196336    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"]
Nov 01 20:25:42.208524 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:25:42.200045    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20853420_c3b3_4956_8e87_be4c6376462f.slice/crio-1050bdd605fc264bc55ab820db440ceccec157b6fdea897be9895ead094449dc WatchSource:0}: Error finding container 1050bdd605fc264bc55ab820db440ceccec157b6fdea897be9895ead094449dc: Status 404 returned error can't find the container with id 1050bdd605fc264bc55ab820db440ceccec157b6fdea897be9895ead094449dc
Nov 01 20:25:42.762328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.762291    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-8514c7"]
Nov 01 20:25:42.765182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.765150    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-8514c7"]
Nov 01 20:25:42.963011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:42.962980    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:43.097177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:43.097085    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl" event={"ID":"20853420-c3b3-4956-8e87-be4c6376462f","Type":"ContainerStarted","Data":"23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd"}
Nov 01 20:25:43.097177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:43.097150    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl" event={"ID":"20853420-c3b3-4956-8e87-be4c6376462f","Type":"ContainerStarted","Data":"1050bdd605fc264bc55ab820db440ceccec157b6fdea897be9895ead094449dc"}
Nov 01 20:25:43.097427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:43.097405    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:43.115383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:43.115322    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl" podStartSLOduration=2.115309615 podStartE2EDuration="2.115309615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:43.111698539 +0000 UTC m=+1588.527331014" watchObservedRunningTime="2026-11-01 20:25:43.115309615 +0000 UTC m=+1588.530942123"
Nov 01 20:25:43.257129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:43.250419    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:43.570916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:43.570877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:44.105210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:44.105168    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-8514c7" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" containerName="test-curl-7417-8514c7" containerID="cri-o://4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b" gracePeriod=2
Nov 01 20:25:44.107497 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:25:44.107471    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-8514c7\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:44.199709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:44.199667    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:44.706163 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:25:44.706115    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-8514c7\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:44.853852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:44.853808    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:45.647334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:45.647298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:46.278333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.277626    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:46.432509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.432481    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:46.472449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.470597    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh\" (UniqueName: \"kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh\") pod \"bf0600fc-7107-4aca-90cf-fb44da81d4bc\" (UID: \"bf0600fc-7107-4aca-90cf-fb44da81d4bc\") "
Nov 01 20:25:46.473374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.473336    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh" pod "bf0600fc-7107-4aca-90cf-fb44da81d4bc" (UID: "bf0600fc-7107-4aca-90cf-fb44da81d4bc"). InnerVolumeSpecName "kube-api-access-r2dgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:46.573399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.573352    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2dgh\" (UniqueName: \"kubernetes.io/projected/bf0600fc-7107-4aca-90cf-fb44da81d4bc-kube-api-access-r2dgh\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:46.710575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.710535    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" path="/var/lib/kubelet/pods/bf0600fc-7107-4aca-90cf-fb44da81d4bc/volumes"
Nov 01 20:25:46.841608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:46.841574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:47.130053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:47.129969    6479 generic.go:356] "Generic (PLEG): container finished" podID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" containerID="4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b" exitCode=137
Nov 01 20:25:47.130053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:47.130013    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-8514c7"
Nov 01 20:25:47.130053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:47.130040    6479 scope.go:122] "RemoveContainer" containerID="4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b"
Nov 01 20:25:47.139958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:47.139940    6479 scope.go:122] "RemoveContainer" containerID="4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b"
Nov 01 20:25:47.140270 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:25:47.140242    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b\": container with ID starting with 4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b not found: ID does not exist" containerID="4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b"
Nov 01 20:25:47.140391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:47.140272    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b"} err="failed to get container status \"4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b\": rpc error: code = NotFound desc = could not find container \"4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b\": container with ID starting with 4927342fca58600aa1694360ddb91d97e74438f8b4526e040c09d92cf6b7ec0b not found: ID does not exist"
Nov 01 20:25:47.528693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:47.528638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:48.164051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:48.164018    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:48.365153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:48.365107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:48.758080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:48.758033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:49.343782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:49.343742    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:49.962416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:49.962382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:50.630535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:50.630500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:51.235278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:51.235239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:51.862057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:51.862026    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:52.443094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:52.443059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:53.071773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:53.071738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:53.662849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:53.662813    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:54.108200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:54.108169    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:25:54.345410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:54.345373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:55.034482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:55.034446    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:55.639142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:55.639085    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:56.235029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:56.234988    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:56.958838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:56.958798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:57.064109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:57.061516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:57.572055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:57.571989    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:58.305783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:58.305704    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:58.870047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:58.870014    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:59.173384 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:25:59.173287    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:25:59.439235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:59.439161    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:25:59.839150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:25:59.839114    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:26:00.191901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:00.191806    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:01.216483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:01.216452    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:01.939970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:01.939932    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:02.742343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:02.742313    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:03.431617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:03.431586    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:04.172675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:04.172245    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:04.836403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:04.836363    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:05.560461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:05.560416    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:06.158838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:06.158797    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:06.928938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:06.928882    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:07.435797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:07.435757    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:08.037231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:08.037197    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:08.652170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:08.652119    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:09.246176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:09.246139    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:09.863835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:09.863796    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:10.462575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:10.462540    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:11.052826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:11.052786    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:11.734779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:11.734746    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:12.350348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:12.350309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:12.952433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:12.952387    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:13.568400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:13.568357    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:14.171841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:14.171802    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:14.685348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:14.685297    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"]
Nov 01 20:26:14.685578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:14.685549    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy" containerID="cri-o://47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3" gracePeriod=30
Nov 01 20:26:14.770814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:14.770777    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:15.425385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:15.425348    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:16.036090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:16.036058    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:16.770304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:16.770266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:17.450508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:17.450474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:18.044257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:18.044222    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:18.735117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:18.735056    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:19.366114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:19.366074    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:19.978325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:19.978298    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:26:20.070092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070057    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070148    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070187    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070267    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070340    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070402    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070446    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070489    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070527    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket\") pod \"28deb188-880a-499b-8934-ad67970c7856\" (UID: \"28deb188-880a-499b-8934-ad67970c7856\") "
Nov 01 20:26:20.070913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.070884    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:20.071153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.071127    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:20.071365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.071333    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:20.071447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.071411    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28deb188-880a-499b-8934-ad67970c7856-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.071447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.071435    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.071564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.071521    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:20.071742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.071711    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:20.072811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.072785    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:20.072913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.072878    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:26:20.073562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.073534    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "kube-api-access-kh75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:20.073862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.073842    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:20.074440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.074421    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token" pod "28deb188-880a-499b-8934-ad67970c7856" (UID: "28deb188-880a-499b-8934-ad67970c7856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:20.178802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178764    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.178802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178800    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.179042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178814    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.179042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178829    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.179042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178843    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh75t\" (UniqueName: \"kubernetes.io/projected/28deb188-880a-499b-8934-ad67970c7856-kube-api-access-kh75t\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.179042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178856    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28deb188-880a-499b-8934-ad67970c7856-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.179042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.178869    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28deb188-880a-499b-8934-ad67970c7856-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:20.514711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.514676    6479 generic.go:356] "Generic (PLEG): container finished" podID="28deb188-880a-499b-8934-ad67970c7856" containerID="47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3" exitCode=0
Nov 01 20:26:20.515152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.514736    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"
Nov 01 20:26:20.515152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.514749    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" event={"ID":"28deb188-880a-499b-8934-ad67970c7856","Type":"ContainerDied","Data":"47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3"}
Nov 01 20:26:20.515152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.514792    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk" event={"ID":"28deb188-880a-499b-8934-ad67970c7856","Type":"ContainerDied","Data":"534bd32b9627460d406e9c7cdd3259c171a396afb8033270106c3b6e660a914a"}
Nov 01 20:26:20.515152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.514816    6479 scope.go:122] "RemoveContainer" containerID="47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3"
Nov 01 20:26:20.529864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.529837    6479 scope.go:122] "RemoveContainer" containerID="b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"
Nov 01 20:26:20.539695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.539670    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"]
Nov 01 20:26:20.540623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.540594    6479 scope.go:122] "RemoveContainer" containerID="47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3"
Nov 01 20:26:20.540951 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:26:20.540927    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3\": container with ID starting with 47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3 not found: ID does not exist" containerID="47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3"
Nov 01 20:26:20.541040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.540958    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3"} err="failed to get container status \"47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3\": rpc error: code = NotFound desc = could not find container \"47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3\": container with ID starting with 47750915840a36a46233f4ce05d905efd4f7f2c5fdd71176c82d62ca88d38db3 not found: ID does not exist"
Nov 01 20:26:20.541040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.540976    6479 scope.go:122] "RemoveContainer" containerID="b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"
Nov 01 20:26:20.541276 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:26:20.541244    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21\": container with ID starting with b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21 not found: ID does not exist" containerID="b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"
Nov 01 20:26:20.541349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.541273    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21"} err="failed to get container status \"b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21\": rpc error: code = NotFound desc = could not find container \"b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21\": container with ID starting with b8f723dfd1fff4f255f8b97db63bd03090db3d7d0ca81cca57dddcfa641bcf21 not found: ID does not exist"
Nov 01 20:26:20.545974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.545948    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-z6grk"]
Nov 01 20:26:20.709419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:20.709384    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28deb188-880a-499b-8934-ad67970c7856" path="/var/lib/kubelet/pods/28deb188-880a-499b-8934-ad67970c7856/volumes"
Nov 01 20:26:21.134097 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:21.134059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:21.453949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:21.453915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:21.554307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:21.554269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:21.857291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:21.857206    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:22.547513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:22.547485    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:22.776136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:22.776097    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:23.117253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:23.117209    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:23.752016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:23.751977    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:24.400860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:24.400811    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:24.515335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:24.515299    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"]
Nov 01 20:26:24.530069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:24.530038    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"]
Nov 01 20:26:24.601074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:24.601042    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:24.704776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:24.704742    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:24.974688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:24.974561    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:25.466978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:25.466940    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:25.833022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:25.832940    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:26.156355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:26.156269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:26.857347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:26.857310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:27.460445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:27.460405    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:28.217499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:28.217463    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:28.814199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:28.814166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:29.172112 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:26:29.172027    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:26:29.465494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.465459    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:29.516198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.516158    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v" podUID="3d12e3c4-b878-4c26-8e0f-7a517b00de25" containerName="payload-processing" containerID="cri-o://a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711" gracePeriod=25
Nov 01 20:26:29.537059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.533749    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk" podUID="797d99ae-3ddb-4af7-a167-cbfd44181dde" containerName="payload-pre-processing" containerID="cri-o://8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d" gracePeriod=25
Nov 01 20:26:29.831443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.831419    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:26:29.883390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.883356    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:26:29.958060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958022    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume\") pod \"797d99ae-3ddb-4af7-a167-cbfd44181dde\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") "
Nov 01 20:26:29.958279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958134    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp\" (UniqueName: \"kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp\") pod \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") "
Nov 01 20:26:29.958279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958161    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume\") pod \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\" (UID: \"3d12e3c4-b878-4c26-8e0f-7a517b00de25\") "
Nov 01 20:26:29.958279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958218    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq\" (UniqueName: \"kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq\") pod \"797d99ae-3ddb-4af7-a167-cbfd44181dde\" (UID: \"797d99ae-3ddb-4af7-a167-cbfd44181dde\") "
Nov 01 20:26:29.958523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958494    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume" pod "797d99ae-3ddb-4af7-a167-cbfd44181dde" (UID: "797d99ae-3ddb-4af7-a167-cbfd44181dde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:29.958608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958559    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume" pod "3d12e3c4-b878-4c26-8e0f-7a517b00de25" (UID: "3d12e3c4-b878-4c26-8e0f-7a517b00de25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:29.958836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958809    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d12e3c4-b878-4c26-8e0f-7a517b00de25-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:29.958968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.958838    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797d99ae-3ddb-4af7-a167-cbfd44181dde-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:29.960353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.960330    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq" pod "797d99ae-3ddb-4af7-a167-cbfd44181dde" (UID: "797d99ae-3ddb-4af7-a167-cbfd44181dde"). InnerVolumeSpecName "kube-api-access-cdqgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:29.960353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:29.960342    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp" pod "3d12e3c4-b878-4c26-8e0f-7a517b00de25" (UID: "3d12e3c4-b878-4c26-8e0f-7a517b00de25"). InnerVolumeSpecName "kube-api-access-z2qbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:30.060812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.060738    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2qbp\" (UniqueName: \"kubernetes.io/projected/3d12e3c4-b878-4c26-8e0f-7a517b00de25-kube-api-access-z2qbp\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:30.060812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.060766    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdqgq\" (UniqueName: \"kubernetes.io/projected/797d99ae-3ddb-4af7-a167-cbfd44181dde-kube-api-access-cdqgq\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:30.205919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.205888    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:30.305949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.305848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:30.613750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.613716    6479 generic.go:356] "Generic (PLEG): container finished" podID="3d12e3c4-b878-4c26-8e0f-7a517b00de25" containerID="a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711" exitCode=0
Nov 01 20:26:30.614235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.613766    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"
Nov 01 20:26:30.614235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.613803    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v" event={"ID":"3d12e3c4-b878-4c26-8e0f-7a517b00de25","Type":"ContainerDied","Data":"a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711"}
Nov 01 20:26:30.614235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.613865    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v" event={"ID":"3d12e3c4-b878-4c26-8e0f-7a517b00de25","Type":"ContainerDied","Data":"0032b4e4c29a302816ccb225be6ff474abbb3fb67f82b90260dfceba85ff060e"}
Nov 01 20:26:30.614235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.613903    6479 scope.go:122] "RemoveContainer" containerID="a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711"
Nov 01 20:26:30.615501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.615479    6479 generic.go:356] "Generic (PLEG): container finished" podID="797d99ae-3ddb-4af7-a167-cbfd44181dde" containerID="8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d" exitCode=0
Nov 01 20:26:30.615636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.615515    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk" event={"ID":"797d99ae-3ddb-4af7-a167-cbfd44181dde","Type":"ContainerDied","Data":"8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d"}
Nov 01 20:26:30.615636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.615542    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"
Nov 01 20:26:30.615636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.615577    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk" event={"ID":"797d99ae-3ddb-4af7-a167-cbfd44181dde","Type":"ContainerDied","Data":"e4acd13d8166ff81538c20f5c18040ce33e4f25ab9267ae1bfd038e0bededbec"}
Nov 01 20:26:30.625022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.624964    6479 scope.go:122] "RemoveContainer" containerID="a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711"
Nov 01 20:26:30.625309 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:26:30.625281    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711\": container with ID starting with a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711 not found: ID does not exist" containerID="a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711"
Nov 01 20:26:30.625378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.625320    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711"} err="failed to get container status \"a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711\": rpc error: code = NotFound desc = could not find container \"a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711\": container with ID starting with a5dcc3c0779b1fa5565ea43745aeb1d04d84af0ff5257adb86a9f894c2d8c711 not found: ID does not exist"
Nov 01 20:26:30.625378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.625347    6479 scope.go:122] "RemoveContainer" containerID="8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d"
Nov 01 20:26:30.636372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.636353    6479 scope.go:122] "RemoveContainer" containerID="8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d"
Nov 01 20:26:30.636727 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:26:30.636702    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d\": container with ID starting with 8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d not found: ID does not exist" containerID="8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d"
Nov 01 20:26:30.636808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.636737    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d"} err="failed to get container status \"8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d\": rpc error: code = NotFound desc = could not find container \"8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d\": container with ID starting with 8120f6bd1b6c7be052e6ae75c014ee7636947df44ab8090d81a3ee28eac4b83d not found: ID does not exist"
Nov 01 20:26:30.638746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.638724    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"]
Nov 01 20:26:30.641436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.641413    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2babaef6-67968898fc-qnp8v"]
Nov 01 20:26:30.651047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.650996    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"]
Nov 01 20:26:30.654020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.654002    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2babaef6-5c67ccc847-tnmtk"]
Nov 01 20:26:30.709337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.709303    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d12e3c4-b878-4c26-8e0f-7a517b00de25" path="/var/lib/kubelet/pods/3d12e3c4-b878-4c26-8e0f-7a517b00de25/volumes"
Nov 01 20:26:30.709743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:30.709729    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797d99ae-3ddb-4af7-a167-cbfd44181dde" path="/var/lib/kubelet/pods/797d99ae-3ddb-4af7-a167-cbfd44181dde/volumes"
Nov 01 20:26:31.066662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:31.066625    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:31.698082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:31.698044    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:31.888670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:31.883271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:32.088070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:32.087981    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:32.699415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:32.699378    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:33.331199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:33.331164    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:33.977420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:33.977377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:34.815399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:34.815361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:34.925287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:34.925230    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:35.458301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:35.458237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:36.140964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:36.140926    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:36.777430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:36.777393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:37.234190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:37.234155    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:26:37.550386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:37.550308    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:37.662423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:37.662377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:37.894733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:37.894436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:38.002675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:38.000179    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:38.807921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:38.807883    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:38.926097 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:38.926057    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:39.187861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:39.187771    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:39.715268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:39.715237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:40.537667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:40.537269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:41.133848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:41.133809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:42.310962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:42.310923    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:43.001291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:43.001253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:43.734568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:43.734510    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:44.455511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:44.455465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:45.139138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:45.139105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:45.761135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:45.761088    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:46.362424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:46.362379    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:46.967741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:46.967706    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:47.544453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:47.544418    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:48.311170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:48.311120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:48.857953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:48.857917    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:49.453434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:49.453400    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:50.038325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:50.038284    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:50.631523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:50.631489    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:50.707055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:50.707017    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:50.808081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:50.808046    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:50.910456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:50.910330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:51.648365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:51.648322    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:51.744713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:51.744635    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:51.868302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:51.868260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:52.756707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:52.756642    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:53.763871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:53.763833    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:54.823149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:54.823100    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:56.012707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:56.010629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:57.070684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:57.068419    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:57.841078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:57.841033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:58.000148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:58.000108    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:58.085736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:58.085696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:58.565056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:58.565016    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:26:59.170940 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:26:59.170900    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:26:59.342999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:26:59.342952    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:00.059675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:00.059614    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:00.843511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:00.843480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:01.475812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:01.475770    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:02.144159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:02.144127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:02.842142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:02.842106    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:03.351708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:03.351671    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:03.468239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:03.468201    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:04.038026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:04.037995    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:04.603976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:04.603929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:04.705804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:04.705762    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:05.323566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:05.323528    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:05.916906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:05.916871    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:27:06.025274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:06.025156    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:07.109528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:07.109489    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:08.037040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:08.037002    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:09.362592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:09.362557    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:09.580585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:09.580546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:10.493335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:10.493289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:11.169498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:11.169466    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:11.751879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:11.751850    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:12.426550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:12.426509    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:12.963130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:12.963092    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:13.098847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:13.098815    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:13.198950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:13.198915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:13.635323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:13.635291    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:14.270224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:14.270184    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:14.877524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:14.877480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:15.559731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:15.559690    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:16.142291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:16.142245    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:16.730338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:16.730307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:17.333486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:17.333448    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:17.964388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:17.964353    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:18.657196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:18.657094    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:19.342932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:19.342902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:20.028099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:20.027638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:20.631675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:20.631606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:21.232105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:21.232068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:21.826733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:21.826696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:22.454623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:22.454588    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:22.816914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:22.816805    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:23.064136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:23.064093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:23.651103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:23.651057    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:24.267623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:24.267592    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:24.837870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:24.837833    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:25.428398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:25.428367    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:26.066479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:26.066445    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:27.013347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:27.013309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:27.735470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:27.735436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:28.340918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:28.340877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:29.074350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:29.074312    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:29.192583 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:27:29.192541    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:27:29.657163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:29.657120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:30.275356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:30.275312    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:30.937619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:30.937582    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:31.600403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:31.600368    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:32.183915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:32.183871    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:32.854786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:32.854746    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:33.436015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:33.435972    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:34.061712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:34.061672    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:34.763691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:34.763637    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:35.358251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:35.358206    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:36.006959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:36.006913    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:36.615495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:36.615453    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:37.196914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:37.196874    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:37.877957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:37.877923    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:38.564583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:38.564532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:39.137740 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:39.137578    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:39.266491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:39.266456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:39.364620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:39.364583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:40.057774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:40.057739    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:40.663598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:40.663566    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:40.837917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:40.837882    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:27:41.455438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:41.455397    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:42.155213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:42.155178    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:43.054124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:43.054088    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:43.891263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:43.891232    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:44.534502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:44.534465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:45.148505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:45.148468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:45.761886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:45.761853    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:46.378329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:46.378295    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:47.176162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:47.176125    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:47.939395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:47.939357    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:48.548673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:48.548618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:49.140833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:49.140788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:49.741856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:49.741809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:50.451258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:50.451220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:50.758719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:50.758619    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"]
Nov 01 20:27:50.761497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:50.761409    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:50.858515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:50.858482    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:51.323579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:51.323545    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:51.437669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:51.437541    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:51.961304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:51.961269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:52.556766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:52.556733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:53.206008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:53.205969    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:53.732835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:53.732801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:54.458685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:54.458631    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:55.138133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:55.138103    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:55.760299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:55.760241    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl" podUID="20853420-c3b3-4956-8e87-be4c6376462f" containerName="payload-processing" containerID="cri-o://23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd" gracePeriod=25
Nov 01 20:27:55.762168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:55.762138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:56.038784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.038698    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:27:56.135624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.135591    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume\") pod \"20853420-c3b3-4956-8e87-be4c6376462f\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") "
Nov 01 20:27:56.135805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.135742    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials\") pod \"20853420-c3b3-4956-8e87-be4c6376462f\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") "
Nov 01 20:27:56.135877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.135806    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume\") pod \"20853420-c3b3-4956-8e87-be4c6376462f\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") "
Nov 01 20:27:56.135962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.135946    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc\") pod \"20853420-c3b3-4956-8e87-be4c6376462f\" (UID: \"20853420-c3b3-4956-8e87-be4c6376462f\") "
Nov 01 20:27:56.136000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.135950    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume" pod "20853420-c3b3-4956-8e87-be4c6376462f" (UID: "20853420-c3b3-4956-8e87-be4c6376462f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:27:56.136241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.136220    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume" pod "20853420-c3b3-4956-8e87-be4c6376462f" (UID: "20853420-c3b3-4956-8e87-be4c6376462f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:27:56.136480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.136454    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:56.136545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.136480    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20853420-c3b3-4956-8e87-be4c6376462f-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:56.138201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.138179    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials" pod "20853420-c3b3-4956-8e87-be4c6376462f" (UID: "20853420-c3b3-4956-8e87-be4c6376462f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:27:56.138201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.138195    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc" pod "20853420-c3b3-4956-8e87-be4c6376462f" (UID: "20853420-c3b3-4956-8e87-be4c6376462f"). InnerVolumeSpecName "kube-api-access-stvkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:27:56.237743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.237710    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvkc\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-kube-api-access-stvkc\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:56.237743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.237738    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20853420-c3b3-4956-8e87-be4c6376462f-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:56.369041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.368932    6479 generic.go:356] "Generic (PLEG): container finished" podID="20853420-c3b3-4956-8e87-be4c6376462f" containerID="23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd" exitCode=0
Nov 01 20:27:56.369041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.368988    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"
Nov 01 20:27:56.369304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.369143    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl" event={"ID":"20853420-c3b3-4956-8e87-be4c6376462f","Type":"ContainerDied","Data":"23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd"}
Nov 01 20:27:56.369304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.369190    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl" event={"ID":"20853420-c3b3-4956-8e87-be4c6376462f","Type":"ContainerDied","Data":"1050bdd605fc264bc55ab820db440ceccec157b6fdea897be9895ead094449dc"}
Nov 01 20:27:56.369304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.369219    6479 scope.go:122] "RemoveContainer" containerID="23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd"
Nov 01 20:27:56.385519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.385487    6479 scope.go:122] "RemoveContainer" containerID="23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd"
Nov 01 20:27:56.385919 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:27:56.385896    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd\": container with ID starting with 23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd not found: ID does not exist" containerID="23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd"
Nov 01 20:27:56.386018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.385943    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd"} err="failed to get container status \"23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd\": rpc error: code = NotFound desc = could not find container \"23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd\": container with ID starting with 23326341e2b1085c405a16577d53be047baeceec9f2017e1fc50117e0c1901cd not found: ID does not exist"
Nov 01 20:27:56.387579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.386945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:56.413680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.413510    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"]
Nov 01 20:27:56.437671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.432956    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-24b244/payload-processing-external-model-e2e-extmod-a-24b244-7d79vp7xl"]
Nov 01 20:27:56.709783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.709743    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20853420-c3b3-4956-8e87-be4c6376462f" path="/var/lib/kubelet/pods/20853420-c3b3-4956-8e87-be4c6376462f/volumes"
Nov 01 20:27:56.968404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:56.968304    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:57.605186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:57.605157    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:57.705228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:57.705197    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:58.075017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:58.074984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:58.370579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:58.370489    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:58.933328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:58.933279    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:27:59.171000 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:27:59.170956    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:27:59.580532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:27:59.580500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:00.213352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:00.213311    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:00.852462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:00.852421    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:01.461420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:01.461382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:02.254725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:02.254689    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:03.140478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:03.139301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:03.256292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:03.256256    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"]
Nov 01 20:28:03.268505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:03.268468    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"]
Nov 01 20:28:03.317681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:03.317584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:03.417074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:03.416981    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:03.940956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:03.940917    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:04.365311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:04.365275    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:04.533676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:04.533627    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"]
Nov 01 20:28:04.533925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:04.533899    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn" podUID="4a4eae91-2776-4013-b4b1-48b3356b1b61" containerName="maas-api" containerID="cri-o://59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba" gracePeriod=30
Nov 01 20:28:04.561621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:04.561583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:04.775294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:04.775236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:05.032109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.032082    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:28:05.035246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.035217    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca\") pod \"4a4eae91-2776-4013-b4b1-48b3356b1b61\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") "
Nov 01 20:28:05.035383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.035288    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs\") pod \"4a4eae91-2776-4013-b4b1-48b3356b1b61\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") "
Nov 01 20:28:05.035430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.035414    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn\" (UniqueName: \"kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn\") pod \"4a4eae91-2776-4013-b4b1-48b3356b1b61\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") "
Nov 01 20:28:05.035467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.035439    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls\") pod \"4a4eae91-2776-4013-b4b1-48b3356b1b61\" (UID: \"4a4eae91-2776-4013-b4b1-48b3356b1b61\") "
Nov 01 20:28:05.035613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.035585    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca" pod "4a4eae91-2776-4013-b4b1-48b3356b1b61" (UID: "4a4eae91-2776-4013-b4b1-48b3356b1b61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:05.035851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.035835    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4eae91-2776-4013-b4b1-48b3356b1b61-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:05.037381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.037354    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs" pod "4a4eae91-2776-4013-b4b1-48b3356b1b61" (UID: "4a4eae91-2776-4013-b4b1-48b3356b1b61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:05.037502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.037477    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn" pod "4a4eae91-2776-4013-b4b1-48b3356b1b61" (UID: "4a4eae91-2776-4013-b4b1-48b3356b1b61"). InnerVolumeSpecName "kube-api-access-9vbvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:05.037597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.037579    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls" pod "4a4eae91-2776-4013-b4b1-48b3356b1b61" (UID: "4a4eae91-2776-4013-b4b1-48b3356b1b61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:05.137584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.137547    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbvn\" (UniqueName: \"kubernetes.io/projected/4a4eae91-2776-4013-b4b1-48b3356b1b61-kube-api-access-9vbvn\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:05.137584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.137580    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:05.137584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.137592    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4eae91-2776-4013-b4b1-48b3356b1b61-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:05.440709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.440592    6479 generic.go:356] "Generic (PLEG): container finished" podID="4a4eae91-2776-4013-b4b1-48b3356b1b61" containerID="59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba" exitCode=0
Nov 01 20:28:05.440709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.440661    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"
Nov 01 20:28:05.440709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.440674    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn" event={"ID":"4a4eae91-2776-4013-b4b1-48b3356b1b61","Type":"ContainerDied","Data":"59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba"}
Nov 01 20:28:05.441249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.440759    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn" event={"ID":"4a4eae91-2776-4013-b4b1-48b3356b1b61","Type":"ContainerDied","Data":"3df09012963a702af2e2054d49f9690c05eb919e26fd01520b3548abeed10992"}
Nov 01 20:28:05.441249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.440783    6479 scope.go:122] "RemoveContainer" containerID="59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba"
Nov 01 20:28:05.444633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.444601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:05.452051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.452028    6479 scope.go:122] "RemoveContainer" containerID="59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba"
Nov 01 20:28:05.452394 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:05.452372    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba\": container with ID starting with 59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba not found: ID does not exist" containerID="59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba"
Nov 01 20:28:05.452507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.452404    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba"} err="failed to get container status \"59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba\": rpc error: code = NotFound desc = could not find container \"59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba\": container with ID starting with 59c46304015a9c6478e0baab57d191e692de3837c2e752b0117b3d7d389832ba not found: ID does not exist"
Nov 01 20:28:05.465328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.465298    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"]
Nov 01 20:28:05.469393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:05.469371    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24b244-5d469c49d5-59dxn"]
Nov 01 20:28:06.065127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:06.065090    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:06.654636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:06.654596    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:06.712630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:06.712595    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4eae91-2776-4013-b4b1-48b3356b1b61" path="/var/lib/kubelet/pods/4a4eae91-2776-4013-b4b1-48b3356b1b61/volumes"
Nov 01 20:28:07.239409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:07.239373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:07.932118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:07.932084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:08.257434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:08.257378    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerName="payload-processing" containerID="cri-o://92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce" gracePeriod=25
Nov 01 20:28:08.269003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:08.268961    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerName="payload-pre-processing" containerID="cri-o://9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6" gracePeriod=25
Nov 01 20:28:08.556252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:08.556144    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:09.120396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:09.120363    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:28:09.543368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:09.543311    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:10.358031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:10.357992    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:11.121340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:11.121302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:11.956236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:11.956198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:12.069921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.067826    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:12.161124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.161079    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:12.377992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.377903    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:12.477672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.477592    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:12.788755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.788708    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"]
Nov 01 20:28:12.789064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.789013    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" containerID="cri-o://4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3" gracePeriod=30
Nov 01 20:28:12.814131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.814074    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:12.925951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:12.925915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:13.402933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:13.402895    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:13.464043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:13.464004    6479 patch_prober.go:29] interesting pod/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:28:13.464205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:13.464071    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:28:13.541436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:13.541401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:13.666304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:13.666226    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:14.334227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:14.334184    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:15.110015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:15.109982    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:15.714535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:15.714494    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:16.298217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:16.298178    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:16.869777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:16.869744    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:17.460505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:17.460463    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:17.863564 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:17.863528    6479 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf15910b_5a7e_4a09_9ea2_26ba8365cd67.slice/crio-4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf15910b_5a7e_4a09_9ea2_26ba8365cd67.slice/crio-conmon-4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 20:28:18.057393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.057316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:18.074881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.074857    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:28:18.237233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237201    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237264    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237303    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237385    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237405    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237434    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237517    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237543    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.237677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237535    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:18.237677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237607    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:18.237677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237620    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:18.238008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237641    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo\") pod \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\" (UID: \"bf15910b-5a7e-4a09-9ea2-26ba8365cd67\") "
Nov 01 20:28:18.238008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237732    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:18.238008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.237949    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:18.238319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.238294    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.238397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.238324    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.238397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.238341    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.238397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.238356    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.238397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.238370    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.239846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.239808    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:18.239846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.239815    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "kube-api-access-2497q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:18.240086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.240069    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:18.240150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.240120    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo" pod "bf15910b-5a7e-4a09-9ea2-26ba8365cd67" (UID: "bf15910b-5a7e-4a09-9ea2-26ba8365cd67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:28:18.339218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.339124    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.339218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.339162    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.339218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.339176    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2497q\" (UniqueName: \"kubernetes.io/projected/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-kube-api-access-2497q\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.339218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.339189    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf15910b-5a7e-4a09-9ea2-26ba8365cd67-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:18.559808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.559776    6479 generic.go:356] "Generic (PLEG): container finished" podID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerID="4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3" exitCode=0
Nov 01 20:28:18.560294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.559842    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"
Nov 01 20:28:18.560294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.559857    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" event={"ID":"bf15910b-5a7e-4a09-9ea2-26ba8365cd67","Type":"ContainerDied","Data":"4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3"}
Nov 01 20:28:18.560294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.559927    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr" event={"ID":"bf15910b-5a7e-4a09-9ea2-26ba8365cd67","Type":"ContainerDied","Data":"494f9fbbf67351bbb80c627cbc2f89d30fb93603f0a167f8ae0729e2f8f45191"}
Nov 01 20:28:18.560294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.559961    6479 scope.go:122] "RemoveContainer" containerID="4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3"
Nov 01 20:28:18.570862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.570820    6479 scope.go:122] "RemoveContainer" containerID="ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"
Nov 01 20:28:18.582927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.582907    6479 scope.go:122] "RemoveContainer" containerID="4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3"
Nov 01 20:28:18.583763 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:18.583381    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3\": container with ID starting with 4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3 not found: ID does not exist" containerID="4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3"
Nov 01 20:28:18.583763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.583416    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3"} err="failed to get container status \"4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3\": rpc error: code = NotFound desc = could not find container \"4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3\": container with ID starting with 4e01df3761bba73c0764b10b4178713189775e1b38f90feb3678dd5dc22bece3 not found: ID does not exist"
Nov 01 20:28:18.583763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.583438    6479 scope.go:122] "RemoveContainer" containerID="ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"
Nov 01 20:28:18.584379 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:18.584321    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b\": container with ID starting with ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b not found: ID does not exist" containerID="ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"
Nov 01 20:28:18.584379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.584356    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b"} err="failed to get container status \"ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b\": rpc error: code = NotFound desc = could not find container \"ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b\": container with ID starting with ecc4aa067523844310420b05811245b4f1e0f3047273343d81edcc7b7ecfe29b not found: ID does not exist"
Nov 01 20:28:18.587001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.586976    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"]
Nov 01 20:28:18.589562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.589507    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-24b244-openshift-default-6f9694767c-tsbjr"]
Nov 01 20:28:18.654519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.654491    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:18.711639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:18.711600    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" path="/var/lib/kubelet/pods/bf15910b-5a7e-4a09-9ea2-26ba8365cd67/volumes"
Nov 01 20:28:19.439810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:19.439776    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:19.540067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:19.540019    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:19.639039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:19.639006    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:19.747168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:19.747111    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:20.280938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:20.280906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:20.825156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:20.825127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:21.530699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:21.530638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:22.138843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:22.138813    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:22.378688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:22.378614    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:22.856923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:22.856881    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:22.967375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:22.967330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:23.402969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:23.402933    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:23.458317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:23.458283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:24.061342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:24.061307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:24.736724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:24.736668    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:25.250772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:25.250730    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:25.543432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:25.543338    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:25.946211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:25.946162    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:26.439789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:26.439706    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:26.644571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:26.644534    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:27.273770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:27.273737    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:27.857417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:27.857382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:28.466201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.466169    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:28.602900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.602866    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:28:28.629818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.629793    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:28:28.663711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.663623    6479 generic.go:356] "Generic (PLEG): container finished" podID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerID="92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce" exitCode=0
Nov 01 20:28:28.663907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.663799    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"
Nov 01 20:28:28.664186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.664157    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" event={"ID":"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c","Type":"ContainerDied","Data":"92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce"}
Nov 01 20:28:28.664307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.664214    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j" event={"ID":"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c","Type":"ContainerDied","Data":"33dcbae3e43795bf0021ac565005cd92c7d470887f1822a6ddbf3cc24577aa12"}
Nov 01 20:28:28.664307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.664247    6479 scope.go:122] "RemoveContainer" containerID="92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce"
Nov 01 20:28:28.665790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.665753    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:28.666596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.666565    6479 generic.go:356] "Generic (PLEG): container finished" podID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerID="9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6" exitCode=0
Nov 01 20:28:28.666735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.666627    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" event={"ID":"30e49bc0-8968-4d82-ba88-62b0199fa183","Type":"ContainerDied","Data":"9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6"}
Nov 01 20:28:28.666813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.666747    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw" event={"ID":"30e49bc0-8968-4d82-ba88-62b0199fa183","Type":"ContainerDied","Data":"fcbb8bd2db56a1ef68ffaa3ba6c23ddd23f69773f8ebfb6d4aa97a029f42ad98"}
Nov 01 20:28:28.666884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.666816    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"
Nov 01 20:28:28.677640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.677609    6479 scope.go:122] "RemoveContainer" containerID="92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce"
Nov 01 20:28:28.678027 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:28.678002    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce\": container with ID starting with 92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce not found: ID does not exist" containerID="92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce"
Nov 01 20:28:28.678106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.678038    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce"} err="failed to get container status \"92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce\": rpc error: code = NotFound desc = could not find container \"92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce\": container with ID starting with 92f5f73bdc584239047ced9cf8a5a3d6e5a3310688eb5ce1848b685cfafd70ce not found: ID does not exist"
Nov 01 20:28:28.678106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.678060    6479 scope.go:122] "RemoveContainer" containerID="9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6"
Nov 01 20:28:28.690445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.690424    6479 scope.go:122] "RemoveContainer" containerID="9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6"
Nov 01 20:28:28.690817 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:28.690790    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6\": container with ID starting with 9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6 not found: ID does not exist" containerID="9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6"
Nov 01 20:28:28.690976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.690951    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6"} err="failed to get container status \"9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6\": rpc error: code = NotFound desc = could not find container \"9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6\": container with ID starting with 9a64544a7c5151ffbeb5b6415045942dd4dd17b54f86df24628029f1739d81f6 not found: ID does not exist"
Nov 01 20:28:28.694591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.694569    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume\") pod \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") "
Nov 01 20:28:28.695345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.695322    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume" pod "bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" (UID: "bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:28.695739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.695716    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4\" (UniqueName: \"kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4\") pod \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\" (UID: \"bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c\") "
Nov 01 20:28:28.696259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.696238    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:28.701941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.701913    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4" pod "bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" (UID: "bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c"). InnerVolumeSpecName "kube-api-access-5dmj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:28.796872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.796794    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume\") pod \"30e49bc0-8968-4d82-ba88-62b0199fa183\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") "
Nov 01 20:28:28.797088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.797060    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7\" (UniqueName: \"kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7\") pod \"30e49bc0-8968-4d82-ba88-62b0199fa183\" (UID: \"30e49bc0-8968-4d82-ba88-62b0199fa183\") "
Nov 01 20:28:28.797150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.797123    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume" pod "30e49bc0-8968-4d82-ba88-62b0199fa183" (UID: "30e49bc0-8968-4d82-ba88-62b0199fa183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:28.798149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.798124    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e49bc0-8968-4d82-ba88-62b0199fa183-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:28.798149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.798151    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dmj4\" (UniqueName: \"kubernetes.io/projected/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c-kube-api-access-5dmj4\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:28.799289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.799270    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7" pod "30e49bc0-8968-4d82-ba88-62b0199fa183" (UID: "30e49bc0-8968-4d82-ba88-62b0199fa183"). InnerVolumeSpecName "kube-api-access-flnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:28.899912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.899870    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnn7\" (UniqueName: \"kubernetes.io/projected/30e49bc0-8968-4d82-ba88-62b0199fa183-kube-api-access-flnn7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:28.982841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.982805    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"]
Nov 01 20:28:28.990031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:28.989994    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24b244-56cb8b6df9-4vt6j"]
Nov 01 20:28:29.001549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:29.001515    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"]
Nov 01 20:28:29.010019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:29.009988    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24b244-749754f7d7-lpdhw"]
Nov 01 20:28:29.103761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:29.103590    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:29.178855 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:29.178816    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:28:29.284677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:29.283596    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:29.930166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:29.930132    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:30.619671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:30.615257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:30.708830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:30.708798    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" path="/var/lib/kubelet/pods/30e49bc0-8968-4d82-ba88-62b0199fa183/volumes"
Nov 01 20:28:30.709404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:30.709382    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" path="/var/lib/kubelet/pods/bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c/volumes"
Nov 01 20:28:31.252480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:31.252446    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:31.826610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:31.826574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:31.922254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:31.922221    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:32.264718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:32.264685    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:32.658748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:32.658637    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:32.850952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:32.850918    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:33.032472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:33.032433    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:33.133343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:33.133307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:33.232414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:33.232381    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:33.853471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:33.853425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:34.388426 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.388390    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:34.452193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452150    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"]
Nov 01 20:28:34.452838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452817    6479 cpu_manager.go:414] "containerMap: removing container" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy"
Nov 01 20:28:34.452838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452838    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452869    6479 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerName="payload-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452877    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerName="payload-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452906    6479 cpu_manager.go:414] "containerMap: removing container" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452915    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452936    6479 cpu_manager.go:414] "containerMap: removing container" podUID="4a4eae91-2776-4013-b4b1-48b3356b1b61" containerName="maas-api"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452943    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4eae91-2776-4013-b4b1-48b3356b1b61" containerName="maas-api"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452967    6479 cpu_manager.go:414] "containerMap: removing container" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerName="payload-pre-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452974    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerName="payload-pre-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452988    6479 cpu_manager.go:414] "containerMap: removing container" podUID="797d99ae-3ddb-4af7-a167-cbfd44181dde" containerName="payload-pre-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.452995    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797d99ae-3ddb-4af7-a167-cbfd44181dde" containerName="payload-pre-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453005    6479 cpu_manager.go:414] "containerMap: removing container" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453011    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453019    6479 cpu_manager.go:414] "containerMap: removing container" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453045    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453066    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3d12e3c4-b878-4c26-8e0f-7a517b00de25" containerName="payload-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453073    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d12e3c4-b878-4c26-8e0f-7a517b00de25" containerName="payload-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453088    6479 cpu_manager.go:414] "containerMap: removing container" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" containerName="test-curl-7417-8514c7"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453095    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" containerName="test-curl-7417-8514c7"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453105    6479 cpu_manager.go:414] "containerMap: removing container" podUID="20853420-c3b3-4956-8e87-be4c6376462f" containerName="payload-processing"
Nov 01 20:28:34.453122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453112    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20853420-c3b3-4956-8e87-be4c6376462f" containerName="payload-processing"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453247    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453258    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4eae91-2776-4013-b4b1-48b3356b1b61" containerName="maas-api"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453268    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e49bc0-8968-4d82-ba88-62b0199fa183" containerName="payload-pre-processing"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453274    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="797d99ae-3ddb-4af7-a167-cbfd44181dde" containerName="payload-pre-processing"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453280    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453285    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf15910b-5a7e-4a09-9ea2-26ba8365cd67" containerName="istio-proxy"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453292    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d12e3c4-b878-4c26-8e0f-7a517b00de25" containerName="payload-processing"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453298    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0600fc-7107-4aca-90cf-fb44da81d4bc" containerName="test-curl-7417-8514c7"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453304    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="20853420-c3b3-4956-8e87-be4c6376462f" containerName="payload-processing"
Nov 01 20:28:34.453826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.453316    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbc3283-e2c4-4e76-a1da-8b4bb6a00e1c" containerName="payload-processing"
Nov 01 20:28:34.458313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.458292    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.464042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.464012    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"]
Nov 01 20:28:34.467399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.467211    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d2dc0ad5-dockercfg-h8bkg\""
Nov 01 20:28:34.467399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.467267    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d2dc0ad5\""
Nov 01 20:28:34.488167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.488128    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:34.507799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.507757    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"]
Nov 01 20:28:34.508631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.508608    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="28deb188-880a-499b-8934-ad67970c7856" containerName="istio-proxy"
Nov 01 20:28:34.512319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.512298    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.519511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.519439    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.519668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.519561    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdd9g\" (UniqueName: \"kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g\") pod \"payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.523638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.523611    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"]
Nov 01 20:28:34.630272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.622511    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.630272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.622581    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") " pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.630272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.622615    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pn5\" (UniqueName: \"kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5\") pod \"payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") " pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.630272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.629513    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdd9g\" (UniqueName: \"kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g\") pod \"payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.637274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.637237    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.664721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.661499    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdd9g\" (UniqueName: \"kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g\") pod \"payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.731108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.731070    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") " pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.731286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.731137    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pn5\" (UniqueName: \"kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5\") pod \"payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") " pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.731792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.731767    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") " pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.738883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.738858    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pn5\" (UniqueName: \"kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5\") pod \"payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") " pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.776414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.776375    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:34.828761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.828714    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:34.947699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:34.945574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:35.090896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.090854    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"]
Nov 01 20:28:35.092536 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:28:35.092508    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b63897e_3823_4d01_a306_c9e921bb82fa.slice/crio-8aceb41e8735f036976138038802797021dda365f0f8b97545c49272d4b56928 WatchSource:0}: Error finding container 8aceb41e8735f036976138038802797021dda365f0f8b97545c49272d4b56928: Status 404 returned error can't find the container with id 8aceb41e8735f036976138038802797021dda365f0f8b97545c49272d4b56928
Nov 01 20:28:35.147961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.147928    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"]
Nov 01 20:28:35.153227 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:28:35.153196    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20d8956_a771_4355_ac23_c47ff178585b.slice/crio-41bd2f4d03abbe9a825fb0e5f0ce6f13e623cdd6b522a101524023e8749e69b2 WatchSource:0}: Error finding container 41bd2f4d03abbe9a825fb0e5f0ce6f13e623cdd6b522a101524023e8749e69b2: Status 404 returned error can't find the container with id 41bd2f4d03abbe9a825fb0e5f0ce6f13e623cdd6b522a101524023e8749e69b2
Nov 01 20:28:35.649588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.649548    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:35.753997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.753940    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96" event={"ID":"6b63897e-3823-4d01-a306-c9e921bb82fa","Type":"ContainerStarted","Data":"d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4"}
Nov 01 20:28:35.754195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.754012    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96" event={"ID":"6b63897e-3823-4d01-a306-c9e921bb82fa","Type":"ContainerStarted","Data":"8aceb41e8735f036976138038802797021dda365f0f8b97545c49272d4b56928"}
Nov 01 20:28:35.754195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.754177    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:35.756322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.756296    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq" event={"ID":"e20d8956-a771-4355-ac23-c47ff178585b","Type":"ContainerStarted","Data":"011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617"}
Nov 01 20:28:35.756470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.756343    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq" event={"ID":"e20d8956-a771-4355-ac23-c47ff178585b","Type":"ContainerStarted","Data":"41bd2f4d03abbe9a825fb0e5f0ce6f13e623cdd6b522a101524023e8749e69b2"}
Nov 01 20:28:35.756962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.756943    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:35.775962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.774835    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96" podStartSLOduration=1.774816103 podStartE2EDuration="1.774816103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:35.768991892 +0000 UTC m=+1761.184624367" watchObservedRunningTime="2026-11-01 20:28:35.774816103 +0000 UTC m=+1761.190448576"
Nov 01 20:28:35.787395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:35.787322    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq" podStartSLOduration=1.7873042510000001 podStartE2EDuration="1.787304251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:35.786941392 +0000 UTC m=+1761.202573878" watchObservedRunningTime="2026-11-01 20:28:35.787304251 +0000 UTC m=+1761.202936724"
Nov 01 20:28:36.440930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:36.440895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:37.256002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:37.255967    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:37.999260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:37.999223    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"]
Nov 01 20:28:38.097382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:38.097346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:38.193745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:38.193710    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:38.457131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:38.457093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:38.742560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:38.742474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:39.336723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:39.336688    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:39.936918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:39.936861    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:40.564088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:40.564053    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:41.322721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:41.316746    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:42.073603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:42.073566    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:42.641545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:42.641505    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:42.752218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:42.752180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:42.866490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:42.866431    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:43.529278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:43.529237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:44.118341 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:44.118292    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:44.246197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:44.246163    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:44.767006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:44.766973    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:45.052467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.052375    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"]
Nov 01 20:28:45.066547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.066496    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"]
Nov 01 20:28:45.114585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.114553    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:45.189611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.189563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:45.501975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.501926    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:45.635156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.635121    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:28:45.912198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:45.912105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:46.769581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:46.769546    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:28:47.149561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:47.149458    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:47.753640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:47.753601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:47.772824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:47.772792    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:28:48.334832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:48.334797    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:48.961932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:48.961902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:49.658058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:49.658011    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:50.065260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.065208    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd" podUID="3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" containerName="payload-processing" containerID="cri-o://e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022" gracePeriod=25
Nov 01 20:28:50.066879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.066844    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf" podUID="4cae4a8b-1f12-403c-a008-9a271813df87" containerName="payload-pre-processing" containerID="cri-o://eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286" gracePeriod=25
Nov 01 20:28:50.381559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.381533    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:28:50.415765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.415671    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:28:50.503319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.503284    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q\" (UniqueName: \"kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q\") pod \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") "
Nov 01 20:28:50.503521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.503343    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g\" (UniqueName: \"kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g\") pod \"4cae4a8b-1f12-403c-a008-9a271813df87\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") "
Nov 01 20:28:50.503521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.503417    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume\") pod \"4cae4a8b-1f12-403c-a008-9a271813df87\" (UID: \"4cae4a8b-1f12-403c-a008-9a271813df87\") "
Nov 01 20:28:50.503521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.503443    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume\") pod \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\" (UID: \"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4\") "
Nov 01 20:28:50.503903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.503870    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume" pod "4cae4a8b-1f12-403c-a008-9a271813df87" (UID: "4cae4a8b-1f12-403c-a008-9a271813df87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:50.503991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.503917    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume" pod "3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" (UID: "3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:50.504165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.504138    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cae4a8b-1f12-403c-a008-9a271813df87-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:50.504279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.504169    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:50.505993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.505964    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g" pod "4cae4a8b-1f12-403c-a008-9a271813df87" (UID: "4cae4a8b-1f12-403c-a008-9a271813df87"). InnerVolumeSpecName "kube-api-access-lfd9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:50.506258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.506237    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q" pod "3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" (UID: "3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4"). InnerVolumeSpecName "kube-api-access-2wp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:50.562495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.562452    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:50.606060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.605959    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wp2q\" (UniqueName: \"kubernetes.io/projected/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4-kube-api-access-2wp2q\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:50.606060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:50.605999    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfd9g\" (UniqueName: \"kubernetes.io/projected/4cae4a8b-1f12-403c-a008-9a271813df87-kube-api-access-lfd9g\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:51.004099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.004059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:51.072764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.072729    6479 generic.go:356] "Generic (PLEG): container finished" podID="3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" containerID="e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022" exitCode=0
Nov 01 20:28:51.072964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.072794    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"
Nov 01 20:28:51.072964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.072792    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd" event={"ID":"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4","Type":"ContainerDied","Data":"e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022"}
Nov 01 20:28:51.072964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.072917    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd" event={"ID":"3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4","Type":"ContainerDied","Data":"b628028ba897d5ef6e26178d18610d6c8cd664c817da92feb468504b166cc1b2"}
Nov 01 20:28:51.072964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.072940    6479 scope.go:122] "RemoveContainer" containerID="e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022"
Nov 01 20:28:51.074476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.074452    6479 generic.go:356] "Generic (PLEG): container finished" podID="4cae4a8b-1f12-403c-a008-9a271813df87" containerID="eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286" exitCode=0
Nov 01 20:28:51.074592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.074499    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"
Nov 01 20:28:51.074592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.074548    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf" event={"ID":"4cae4a8b-1f12-403c-a008-9a271813df87","Type":"ContainerDied","Data":"eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286"}
Nov 01 20:28:51.074840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.074598    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf" event={"ID":"4cae4a8b-1f12-403c-a008-9a271813df87","Type":"ContainerDied","Data":"b841102967adcb1c372a2864f59584d416de4ac96aed3615cb479e01075d428f"}
Nov 01 20:28:51.095164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.095137    6479 scope.go:122] "RemoveContainer" containerID="e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022"
Nov 01 20:28:51.095560 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:51.095534    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022\": container with ID starting with e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022 not found: ID does not exist" containerID="e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022"
Nov 01 20:28:51.095676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.095576    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022"} err="failed to get container status \"e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022\": rpc error: code = NotFound desc = could not find container \"e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022\": container with ID starting with e89a65957fa2c524539f0d4ef17510e992379b1e745fde9680d7b586710fc022 not found: ID does not exist"
Nov 01 20:28:51.095676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.095600    6479 scope.go:122] "RemoveContainer" containerID="eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286"
Nov 01 20:28:51.104474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.104444    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"]
Nov 01 20:28:51.107487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.107459    6479 scope.go:122] "RemoveContainer" containerID="eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286"
Nov 01 20:28:51.107901 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:51.107874    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286\": container with ID starting with eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286 not found: ID does not exist" containerID="eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286"
Nov 01 20:28:51.107999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.107915    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286"} err="failed to get container status \"eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286\": rpc error: code = NotFound desc = could not find container \"eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286\": container with ID starting with eaff03ac85264c334c793626fb15b2c9be72a7037d8960f77c201a7cfb00d286 not found: ID does not exist"
Nov 01 20:28:51.109244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.109221    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9de2ac-6b849b9989-xmfwd"]
Nov 01 20:28:51.126395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.126354    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"]
Nov 01 20:28:51.130812 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.130781    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9de2ac-dcb8cdf67-hm9vf"]
Nov 01 20:28:51.156474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.156443    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:51.591691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.591627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:51.809256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:51.809220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:52.396512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:52.396462    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:52.709658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:52.709623    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" path="/var/lib/kubelet/pods/3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4/volumes"
Nov 01 20:28:52.710091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:52.710077    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cae4a8b-1f12-403c-a008-9a271813df87" path="/var/lib/kubelet/pods/4cae4a8b-1f12-403c-a008-9a271813df87/volumes"
Nov 01 20:28:53.022149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.022057    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerName="main" containerID="cri-o://81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0" gracePeriod=45
Nov 01 20:28:53.066003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.065965    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:53.350487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.350454    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:28:53.456243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456206    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache\") pod \"b3dce113-c837-4cd8-bcb9-0a8f41476159\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") "
Nov 01 20:28:53.456243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456249    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home\") pod \"b3dce113-c837-4cd8-bcb9-0a8f41476159\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") "
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456287    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs\") pod \"b3dce113-c837-4cd8-bcb9-0a8f41476159\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") "
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456365    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm\") pod \"b3dce113-c837-4cd8-bcb9-0a8f41476159\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") "
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456515    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir\") pod \"b3dce113-c837-4cd8-bcb9-0a8f41476159\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") "
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456552    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm\" (UniqueName: \"kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm\") pod \"b3dce113-c837-4cd8-bcb9-0a8f41476159\" (UID: \"b3dce113-c837-4cd8-bcb9-0a8f41476159\") "
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456563    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache" pod "b3dce113-c837-4cd8-bcb9-0a8f41476159" (UID: "b3dce113-c837-4cd8-bcb9-0a8f41476159"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456576    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home" pod "b3dce113-c837-4cd8-bcb9-0a8f41476159" (UID: "b3dce113-c837-4cd8-bcb9-0a8f41476159"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.456887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.456786    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir" pod "b3dce113-c837-4cd8-bcb9-0a8f41476159" (UID: "b3dce113-c837-4cd8-bcb9-0a8f41476159"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.457317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.457285    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.457386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.457316    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.457386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.457331    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.458637 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.458606    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm" pod "b3dce113-c837-4cd8-bcb9-0a8f41476159" (UID: "b3dce113-c837-4cd8-bcb9-0a8f41476159"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.458761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.458718    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs" pod "b3dce113-c837-4cd8-bcb9-0a8f41476159" (UID: "b3dce113-c837-4cd8-bcb9-0a8f41476159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:53.459020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.459003    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm" pod "b3dce113-c837-4cd8-bcb9-0a8f41476159" (UID: "b3dce113-c837-4cd8-bcb9-0a8f41476159"). InnerVolumeSpecName "kube-api-access-89fbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:53.559193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.559152    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3dce113-c837-4cd8-bcb9-0a8f41476159-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.559193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.559187    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3dce113-c837-4cd8-bcb9-0a8f41476159-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.559193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.559196    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89fbm\" (UniqueName: \"kubernetes.io/projected/b3dce113-c837-4cd8-bcb9-0a8f41476159-kube-api-access-89fbm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.908182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:53.908140    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:54.099055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.099012    6479 generic.go:356] "Generic (PLEG): container finished" podID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerID="81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0" exitCode=0
Nov 01 20:28:54.099239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.099060    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"
Nov 01 20:28:54.099239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.099086    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" event={"ID":"b3dce113-c837-4cd8-bcb9-0a8f41476159","Type":"ContainerDied","Data":"81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0"}
Nov 01 20:28:54.099239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.099133    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" event={"ID":"b3dce113-c837-4cd8-bcb9-0a8f41476159","Type":"ContainerDied","Data":"5990ed54aed69ea24c13cd6d8fdaf59a28884a612f8048f9ddac0d8d77656ea9"}
Nov 01 20:28:54.099239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.099150    6479 scope.go:122] "RemoveContainer" containerID="81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0"
Nov 01 20:28:54.112375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.112354    6479 scope.go:122] "RemoveContainer" containerID="81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0"
Nov 01 20:28:54.112758 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:54.112728    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0\": container with ID starting with 81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0 not found: ID does not exist" containerID="81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0"
Nov 01 20:28:54.112864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.112765    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0"} err="failed to get container status \"81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0\": rpc error: code = NotFound desc = could not find container \"81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0\": container with ID starting with 81ddd3ea1756a90137f3af27bf62f153ad82cb1f3ff49471b97f08b7e59e59f0 not found: ID does not exist"
Nov 01 20:28:54.123587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.123563    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"]
Nov 01 20:28:54.126332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.126308    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp"]
Nov 01 20:28:54.272860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.272820    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/premium-w3-9de2ac-kserve-5d7c9fb99c-jp2bp" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:28:54.709932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.709897    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" path="/var/lib/kubelet/pods/b3dce113-c837-4cd8-bcb9-0a8f41476159/volumes"
Nov 01 20:28:54.755607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.755572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:54.856159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:54.856110    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:55.023110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:55.023019    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:55.658227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:55.658194    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:56.419345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:56.419308    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:56.520071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:56.520032    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:57.042861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:57.042829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:57.594352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:57.594316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:57.714412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:57.714376    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:58.201787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:58.201731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:58.415922 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:58.415881    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:59.015128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:59.015019    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:59.107338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:59.107301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:59.282154 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:28:59.282058    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:28:59.580542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:59.580451    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:28:59.874882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:28:59.874800    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:00.578512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:00.578473    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:01.247287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:01.247250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:01.861294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:01.861255    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:02.362989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:02.362948    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:02.459216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:02.459186    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:02.777742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:02.777707    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:03.518509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:03.518471    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:03.623150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:03.623059    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:04.038767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:04.038719    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:05.132288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:05.132253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:05.970317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:05.970277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:06.228759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:06.228671    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:06.561537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:06.561445    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:07.150194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:07.150157    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:07.744527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:07.744489    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:08.344950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:08.344911    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:08.972600 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:08.972562    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:09.401526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.401410    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"]
Nov 01 20:29:09.402440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.402403    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m" podUID="66b12b04-38d7-4fbd-8b9c-b3de31c6225c" containerName="maas-api" containerID="cri-o://54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61" gracePeriod=30
Nov 01 20:29:09.445538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.445505    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:09.674847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.654191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:09.860230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.860201    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:29:09.952817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.952779    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca\") pod \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") "
Nov 01 20:29:09.953034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.952843    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs\") pod \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") "
Nov 01 20:29:09.953034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.952926    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls\") pod \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") "
Nov 01 20:29:09.953034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.952986    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m\" (UniqueName: \"kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m\") pod \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\" (UID: \"66b12b04-38d7-4fbd-8b9c-b3de31c6225c\") "
Nov 01 20:29:09.953325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.953287    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca" pod "66b12b04-38d7-4fbd-8b9c-b3de31c6225c" (UID: "66b12b04-38d7-4fbd-8b9c-b3de31c6225c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:09.953553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.953531    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:09.955137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.955113    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls" pod "66b12b04-38d7-4fbd-8b9c-b3de31c6225c" (UID: "66b12b04-38d7-4fbd-8b9c-b3de31c6225c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:29:09.955321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.955292    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs" pod "66b12b04-38d7-4fbd-8b9c-b3de31c6225c" (UID: "66b12b04-38d7-4fbd-8b9c-b3de31c6225c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:29:09.955434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:09.955380    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m" pod "66b12b04-38d7-4fbd-8b9c-b3de31c6225c" (UID: "66b12b04-38d7-4fbd-8b9c-b3de31c6225c"). InnerVolumeSpecName "kube-api-access-phf9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:10.055400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.055346    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:10.055400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.055395    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:10.055867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.055410    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phf9m\" (UniqueName: \"kubernetes.io/projected/66b12b04-38d7-4fbd-8b9c-b3de31c6225c-kube-api-access-phf9m\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:10.187180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.187140    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"]
Nov 01 20:29:10.188064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188046    6479 cpu_manager.go:414] "containerMap: removing container" podUID="4cae4a8b-1f12-403c-a008-9a271813df87" containerName="payload-pre-processing"
Nov 01 20:29:10.188135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188067    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cae4a8b-1f12-403c-a008-9a271813df87" containerName="payload-pre-processing"
Nov 01 20:29:10.188135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188091    6479 cpu_manager.go:414] "containerMap: removing container" podUID="66b12b04-38d7-4fbd-8b9c-b3de31c6225c" containerName="maas-api"
Nov 01 20:29:10.188135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188098    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b12b04-38d7-4fbd-8b9c-b3de31c6225c" containerName="maas-api"
Nov 01 20:29:10.188135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188128    6479 cpu_manager.go:414] "containerMap: removing container" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerName="main"
Nov 01 20:29:10.188135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188135    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerName="main"
Nov 01 20:29:10.188290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188165    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" containerName="payload-processing"
Nov 01 20:29:10.188290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188173    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" containerName="payload-processing"
Nov 01 20:29:10.188361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188298    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cae4a8b-1f12-403c-a008-9a271813df87" containerName="payload-pre-processing"
Nov 01 20:29:10.188361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188309    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b12b04-38d7-4fbd-8b9c-b3de31c6225c" containerName="maas-api"
Nov 01 20:29:10.188361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188321    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dce113-c837-4cd8-bcb9-0a8f41476159" containerName="main"
Nov 01 20:29:10.188361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.188336    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bba70c6-6a6b-4cf7-9ac2-bd64f727f5f4" containerName="payload-processing"
Nov 01 20:29:10.192255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.192236    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.199160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.199132    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-036c69-openshift-default-dockercfg-9njcd\""
Nov 01 20:29:10.207547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.207516    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"]
Nov 01 20:29:10.257591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.257550    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.257824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.257620    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.257824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.257742    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.257824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.257801    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.258020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.257909    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qkj\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.258020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.257947    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.258141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.258089    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.258194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.258136    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.258233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.258215    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.298824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.298786    6479 generic.go:356] "Generic (PLEG): container finished" podID="66b12b04-38d7-4fbd-8b9c-b3de31c6225c" containerID="54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61" exitCode=0
Nov 01 20:29:10.299020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.298851    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"
Nov 01 20:29:10.299020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.298849    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m" event={"ID":"66b12b04-38d7-4fbd-8b9c-b3de31c6225c","Type":"ContainerDied","Data":"54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61"}
Nov 01 20:29:10.299020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.298982    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m" event={"ID":"66b12b04-38d7-4fbd-8b9c-b3de31c6225c","Type":"ContainerDied","Data":"b3f88e12f139952d90645d71c9eeaaa11d7e709a9a443e5728b7e11a76bf96fa"}
Nov 01 20:29:10.299020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.299013    6479 scope.go:122] "RemoveContainer" containerID="54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61"
Nov 01 20:29:10.310594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.310565    6479 scope.go:122] "RemoveContainer" containerID="54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61"
Nov 01 20:29:10.310945 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:29:10.310921    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61\": container with ID starting with 54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61 not found: ID does not exist" containerID="54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61"
Nov 01 20:29:10.311020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.310958    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61"} err="failed to get container status \"54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61\": rpc error: code = NotFound desc = could not find container \"54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61\": container with ID starting with 54e988543ca3f02f37bf2b8255079f694c717e4d36077017bfb910288bd6be61 not found: ID does not exist"
Nov 01 20:29:10.322381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.322341    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"]
Nov 01 20:29:10.325736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.325701    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9de2ac-96d7b6c45-wph8m"]
Nov 01 20:29:10.344041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.343993    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:10.359422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359383    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359461    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359502    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359542    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359614    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qkj\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359677    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359787    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.359872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359833    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.360021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359871    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.360021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359878    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.360021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359938    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.360021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.359904    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.360291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.360269    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.360405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.360363    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.362486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.362466    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.362582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.362467    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.367739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.367715    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.367973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.367951    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qkj\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.520277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.520184    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:10.708942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.708908    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b12b04-38d7-4fbd-8b9c-b3de31c6225c" path="/var/lib/kubelet/pods/66b12b04-38d7-4fbd-8b9c-b3de31c6225c/volumes"
Nov 01 20:29:10.807044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.807012    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"]
Nov 01 20:29:10.808586 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:29:10.808554    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d665fd_290f_4d5f_8b56_ad7bbaebeaf9.slice/crio-3874fe046bd71fa4f42a956a6db4fd92f1003b39e4d1185cb15caa7dcf9040de WatchSource:0}: Error finding container 3874fe046bd71fa4f42a956a6db4fd92f1003b39e4d1185cb15caa7dcf9040de: Status 404 returned error can't find the container with id 3874fe046bd71fa4f42a956a6db4fd92f1003b39e4d1185cb15caa7dcf9040de
Nov 01 20:29:10.811796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.811762    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:29:10.811878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.811832    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:29:10.811878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:10.811863    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:29:11.133011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.132921    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:11.316477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.315003    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" event={"ID":"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9","Type":"ContainerStarted","Data":"6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43"}
Nov 01 20:29:11.316477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.315084    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" event={"ID":"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9","Type":"ContainerStarted","Data":"3874fe046bd71fa4f42a956a6db4fd92f1003b39e4d1185cb15caa7dcf9040de"}
Nov 01 20:29:11.346320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.346267    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podStartSLOduration=1.346251407 podStartE2EDuration="1.346251407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:11.342275377 +0000 UTC m=+1796.757907873" watchObservedRunningTime="2026-11-01 20:29:11.346251407 +0000 UTC m=+1796.761883879"
Nov 01 20:29:11.521043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.520997    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:11.522677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.522626    6479 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 01 20:29:11.522814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.522711    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 01 20:29:11.834194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:11.834111    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:12.435132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:12.435098    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:12.534500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:12.533798    6479 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 01 20:29:12.534500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:12.533878    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 01 20:29:13.132401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:13.132363    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:13.520906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:13.520860    6479 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 01 20:29:13.521353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:13.520928    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 01 20:29:13.726596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:13.726559    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:13.841726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:13.841287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:14.141078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:14.140993    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:14.520909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:14.520872    6479 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 01 20:29:14.521130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:14.520922    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 01 20:29:14.522572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:14.522544    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:14.621067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:14.621030    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:15.366707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:15.362061    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:15.520805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:15.520750    6479 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 01 20:29:15.521020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:15.520831    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 01 20:29:15.972671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:15.972275    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:16.520634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:16.520600    6479 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 01 20:29:16.520864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:16.520673    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 01 20:29:16.602949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:16.602905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:17.233094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:17.233057    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:17.567508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:17.567413    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:17.567891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:17.567870    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:17.568894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:17.568870    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:29:17.838428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:17.838340    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:29:17.844978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:17.844952    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:18.430670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:18.426987    6479 scope.go:122] "RemoveContainer" containerID="40780a4d5f6e40dd73a129a49797f408b1d76890acadfc812f24174db40a9c3a"
Nov 01 20:29:18.797736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:18.795476    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:19.350273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:19.350238    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:20.058007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:20.057972    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:20.655386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:20.655341    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:21.359049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:21.359014    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:22.012663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:22.012624    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:22.667844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:22.667803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:22.904300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:22.904250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:23.359412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:23.359356    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:23.942194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:23.942158    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:24.537366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:24.537330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:25.154822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:25.154786    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:25.756117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:25.756074    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:26.361694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:26.361641    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:27.058573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:27.058530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:27.654533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:27.654483    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:28.238054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:28.238023    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:28.835772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:28.835731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:29.173055 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:29:29.172955    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:29:29.441217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:29.441136    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:30.059050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:30.059013    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:30.671169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:30.671134    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:30.987078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:30.987039    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:29:31.256482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:31.256390    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:31.961950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:31.961912    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:32.633432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:32.633355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:33.142100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:33.142057    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:33.731989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:33.731946    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:34.351295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:34.351253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:35.035898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:35.035863    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:35.646500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:35.646460    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:36.234378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:36.234347    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:36.965516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:36.965472    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:37.556230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:37.556191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:38.147885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:38.147849    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:38.977815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:38.977773    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:39.534769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:39.534734    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:40.145497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:40.145454    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:40.839478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:40.839444    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:41.511602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:41.511561    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:42.038105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:42.038062    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:42.631051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:42.631005    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:43.375746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:43.375698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:43.961904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:43.961869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:44.552831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:44.552793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:45.176313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:45.176271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:45.761984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:45.761922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:45.945795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:45.945756    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:29:46.462507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:46.462465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:47.055528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:47.055481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:47.648354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:47.648314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:48.237738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:48.237703    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:48.842277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:48.842237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:49.466400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:49.466363    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:50.072301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:50.072251    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:50.725931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:50.725901    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:51.354837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:51.354770    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:51.951194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:51.951146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:52.547828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:52.547793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:53.146918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:53.146860    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:53.734277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:53.734242    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:54.437256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:54.437221    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:55.227560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:55.227491    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:55.858754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:55.858718    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:56.465166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:56.465122    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:57.051057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:57.051017    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:57.635335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:57.635290    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:58.284182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:58.284146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:58.943943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:58.943905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:29:59.173052 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:29:59.173015    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:29:59.569510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:29:59.569466    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:00.171946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.171911    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"]
Nov 01 20:30:00.177975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.177944    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"]
Nov 01 20:30:00.178164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.178071    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.193623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.193583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:00.212225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.212171    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"]
Nov 01 20:30:00.212544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.212276    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.214776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.214742    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"]
Nov 01 20:30:00.221725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.219856    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"]
Nov 01 20:30:00.244709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.230395    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.306755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.306725    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:30:00.312588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.312495    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"]
Nov 01 20:30:00.334482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.334440    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.334805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.334777    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.335029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.335011    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8vf\" (UniqueName: \"kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.335290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.335269    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5twg7\" (UniqueName: \"kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7\") pod \"maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.335472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.335451    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfw5g\" (UniqueName: \"kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.336003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.335981    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.437277 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.437186    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8vf\" (UniqueName: \"kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.437451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.437329    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twg7\" (UniqueName: \"kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7\") pod \"maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.437451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.437347    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfw5g\" (UniqueName: \"kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.437451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.437384    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.437641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.437460    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.437641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.437505    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.438185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.438156    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.438185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.438176    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.438185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.438169    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.445800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.445773    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twg7\" (UniqueName: \"kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7\") pod \"maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.446350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.446324    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8vf\" (UniqueName: \"kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.446461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.446447    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfw5g\" (UniqueName: \"kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.506296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.506252    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:00.634047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.634011    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:00.645769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.645741    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:00.795073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.794989    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"]
Nov 01 20:30:00.798677 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:30:00.798633    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba424b9_8911_4392_81ae_d68379ef9f13.slice/crio-b2c733af0cd8cf8552297e1a2248f5f6ebd5ed2590e41c9c40929a6503717cb1 WatchSource:0}: Error finding container b2c733af0cd8cf8552297e1a2248f5f6ebd5ed2590e41c9c40929a6503717cb1: Status 404 returned error can't find the container with id b2c733af0cd8cf8552297e1a2248f5f6ebd5ed2590e41c9c40929a6503717cb1
Nov 01 20:30:00.924478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.924446    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"]
Nov 01 20:30:00.926600 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:30:00.926531    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7e7e77_276b_4c5a_8d4c_2914aa42b9ba.slice/crio-114a6abc0b62bc7ecc3340fed2c4370c5cfb995e81a1dfbb377637a71d5aa1cf WatchSource:0}: Error finding container 114a6abc0b62bc7ecc3340fed2c4370c5cfb995e81a1dfbb377637a71d5aa1cf: Status 404 returned error can't find the container with id 114a6abc0b62bc7ecc3340fed2c4370c5cfb995e81a1dfbb377637a71d5aa1cf
Nov 01 20:30:00.941786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.941761    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"]
Nov 01 20:30:00.943436 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:30:00.943405    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9146ed68_c9db_48a0_8263_e884d69c3d10.slice/crio-a7d10758c76c86706a4596465b836aa858ca97ed081c08949fcdaf48da2cb8e7 WatchSource:0}: Error finding container a7d10758c76c86706a4596465b836aa858ca97ed081c08949fcdaf48da2cb8e7: Status 404 returned error can't find the container with id a7d10758c76c86706a4596465b836aa858ca97ed081c08949fcdaf48da2cb8e7
Nov 01 20:30:00.956894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:00.956859    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:01.762593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.762540    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:01.783793 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.783756    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerStarted","Data":"39384e5d5f092c4e12d9bff107ed8d725a8c56a0c14cfa63979351b24307c6e3"}
Nov 01 20:30:01.784003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.783818    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerStarted","Data":"a7d10758c76c86706a4596465b836aa858ca97ed081c08949fcdaf48da2cb8e7"}
Nov 01 20:30:01.785440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.785402    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerStarted","Data":"571fdc5943904a92bfb2e28e1c9325c40ff3a4c2720f0d64f355130bc8104def"}
Nov 01 20:30:01.785557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.785459    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerStarted","Data":"b2c733af0cd8cf8552297e1a2248f5f6ebd5ed2590e41c9c40929a6503717cb1"}
Nov 01 20:30:01.787110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.787084    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" event={"ID":"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba","Type":"ContainerStarted","Data":"a782b1d35cab764aa3db50a3f547b2399a3f6bb1e543c61d1c7ea73c169f3a3a"}
Nov 01 20:30:01.787220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.787122    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" event={"ID":"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba","Type":"ContainerStarted","Data":"114a6abc0b62bc7ecc3340fed2c4370c5cfb995e81a1dfbb377637a71d5aa1cf"}
Nov 01 20:30:01.814808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.807548    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" podStartSLOduration=1.807530211 podStartE2EDuration="1.807530211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:01.80316557 +0000 UTC m=+1847.218798042" watchObservedRunningTime="2026-11-01 20:30:01.807530211 +0000 UTC m=+1847.223162691"
Nov 01 20:30:01.887106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.887032    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" podStartSLOduration=1.8870079789999998 podStartE2EDuration="1.887007979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:01.876994708 +0000 UTC m=+1847.292627188" watchObservedRunningTime="2026-11-01 20:30:01.887007979 +0000 UTC m=+1847.302640453"
Nov 01 20:30:01.893885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:01.887975    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" podStartSLOduration=1.887959222 podStartE2EDuration="1.887959222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:01.843292031 +0000 UTC m=+1847.258924520" watchObservedRunningTime="2026-11-01 20:30:01.887959222 +0000 UTC m=+1847.303591699"
Nov 01 20:30:02.378873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.378773    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:02.795236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.795198    6479 generic.go:356] "Generic (PLEG): container finished" podID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerID="39384e5d5f092c4e12d9bff107ed8d725a8c56a0c14cfa63979351b24307c6e3" exitCode=60
Nov 01 20:30:02.795787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.795276    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerDied","Data":"39384e5d5f092c4e12d9bff107ed8d725a8c56a0c14cfa63979351b24307c6e3"}
Nov 01 20:30:02.795787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.795671    6479 scope.go:122] "RemoveContainer" containerID="39384e5d5f092c4e12d9bff107ed8d725a8c56a0c14cfa63979351b24307c6e3"
Nov 01 20:30:02.797530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.797424    6479 generic.go:356] "Generic (PLEG): container finished" podID="cba424b9-8911-4392-81ae-d68379ef9f13" containerID="571fdc5943904a92bfb2e28e1c9325c40ff3a4c2720f0d64f355130bc8104def" exitCode=60
Nov 01 20:30:02.797530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.797521    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerDied","Data":"571fdc5943904a92bfb2e28e1c9325c40ff3a4c2720f0d64f355130bc8104def"}
Nov 01 20:30:02.797842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.797823    6479 scope.go:122] "RemoveContainer" containerID="571fdc5943904a92bfb2e28e1c9325c40ff3a4c2720f0d64f355130bc8104def"
Nov 01 20:30:02.799479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.799455    6479 generic.go:356] "Generic (PLEG): container finished" podID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerID="a782b1d35cab764aa3db50a3f547b2399a3f6bb1e543c61d1c7ea73c169f3a3a" exitCode=60
Nov 01 20:30:02.799572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.799554    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" event={"ID":"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba","Type":"ContainerDied","Data":"a782b1d35cab764aa3db50a3f547b2399a3f6bb1e543c61d1c7ea73c169f3a3a"}
Nov 01 20:30:02.800015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:02.799996    6479 scope.go:122] "RemoveContainer" containerID="a782b1d35cab764aa3db50a3f547b2399a3f6bb1e543c61d1c7ea73c169f3a3a"
Nov 01 20:30:03.103842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.103807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:03.740091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.740051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:03.848502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.847670    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerStarted","Data":"355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466"}
Nov 01 20:30:03.850401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.850364    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerStarted","Data":"fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985"}
Nov 01 20:30:03.853088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.852538    6479 generic.go:356] "Generic (PLEG): container finished" podID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerID="4a90c802a63158a5c9450456040886bd2ded10d5184d0027116c42a1634cc9ee" exitCode=60
Nov 01 20:30:03.853088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.852614    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" event={"ID":"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba","Type":"ContainerDied","Data":"4a90c802a63158a5c9450456040886bd2ded10d5184d0027116c42a1634cc9ee"}
Nov 01 20:30:03.853088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.852686    6479 scope.go:122] "RemoveContainer" containerID="a782b1d35cab764aa3db50a3f547b2399a3f6bb1e543c61d1c7ea73c169f3a3a"
Nov 01 20:30:03.853088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:03.852938    6479 scope.go:122] "RemoveContainer" containerID="4a90c802a63158a5c9450456040886bd2ded10d5184d0027116c42a1634cc9ee"
Nov 01 20:30:03.853719 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:03.853680    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b_odh-ai-gateway-infra(9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba"
Nov 01 20:30:04.498724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.498678    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:04.860768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.860667    6479 generic.go:356] "Generic (PLEG): container finished" podID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerID="355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466" exitCode=60
Nov 01 20:30:04.860768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.860745    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerDied","Data":"355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466"}
Nov 01 20:30:04.861337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.860810    6479 scope.go:122] "RemoveContainer" containerID="39384e5d5f092c4e12d9bff107ed8d725a8c56a0c14cfa63979351b24307c6e3"
Nov 01 20:30:04.861337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.861067    6479 scope.go:122] "RemoveContainer" containerID="355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466"
Nov 01 20:30:04.861469 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:04.861443    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m_odh-ai-gateway-infra(9146ed68-c9db-48a0-8263-e884d69c3d10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10"
Nov 01 20:30:04.863047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.863016    6479 generic.go:356] "Generic (PLEG): container finished" podID="cba424b9-8911-4392-81ae-d68379ef9f13" containerID="fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985" exitCode=60
Nov 01 20:30:04.863164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.863096    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerDied","Data":"fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985"}
Nov 01 20:30:04.863412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.863397    6479 scope.go:122] "RemoveContainer" containerID="fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985"
Nov 01 20:30:04.863768 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:04.863745    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw_odh-ai-gateway-infra(cba424b9-8911-4392-81ae-d68379ef9f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" podUID="cba424b9-8911-4392-81ae-d68379ef9f13"
Nov 01 20:30:04.865454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.865425    6479 scope.go:122] "RemoveContainer" containerID="4a90c802a63158a5c9450456040886bd2ded10d5184d0027116c42a1634cc9ee"
Nov 01 20:30:04.865834 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:04.865808    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b_odh-ai-gateway-infra(9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba"
Nov 01 20:30:04.874283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:04.874260    6479 scope.go:122] "RemoveContainer" containerID="571fdc5943904a92bfb2e28e1c9325c40ff3a4c2720f0d64f355130bc8104def"
Nov 01 20:30:05.160384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:05.160290    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:05.850530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:05.850468    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:05.875655 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:05.875611    6479 scope.go:122] "RemoveContainer" containerID="fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985"
Nov 01 20:30:05.876150 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:05.876011    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw_odh-ai-gateway-infra(cba424b9-8911-4392-81ae-d68379ef9f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" podUID="cba424b9-8911-4392-81ae-d68379ef9f13"
Nov 01 20:30:05.877806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:05.877781    6479 scope.go:122] "RemoveContainer" containerID="355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466"
Nov 01 20:30:05.878149 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:05.878124    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m_odh-ai-gateway-infra(9146ed68-c9db-48a0-8263-e884d69c3d10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10"
Nov 01 20:30:06.508395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:06.508345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:07.143449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:07.143410    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:07.741280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:07.741246    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:08.346275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:08.346237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:08.959784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:08.959745    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:09.563392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:09.563335    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:10.163170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:10.163115    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:10.754114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:10.754074    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:11.353002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:11.352965    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:11.930298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:11.930261    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:12.529404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:12.529364    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:13.135324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:13.135285    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:13.734297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:13.734257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:14.518970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:14.518931    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:14.706630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:14.706601    6479 scope.go:122] "RemoveContainer" containerID="355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466"
Nov 01 20:30:14.754352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:14.754315    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:14.855986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:14.855901    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:15.537912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:15.537876    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:15.700761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:15.700727    6479 scope.go:122] "RemoveContainer" containerID="4a90c802a63158a5c9450456040886bd2ded10d5184d0027116c42a1634cc9ee"
Nov 01 20:30:15.944682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:15.944617    6479 generic.go:356] "Generic (PLEG): container finished" podID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerID="d6ddfc131c557997af4fb04e8080b42e84098a0f72ad3c205716b26c3844f2e6" exitCode=60
Nov 01 20:30:15.944880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:15.944710    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerDied","Data":"d6ddfc131c557997af4fb04e8080b42e84098a0f72ad3c205716b26c3844f2e6"}
Nov 01 20:30:15.944880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:15.944786    6479 scope.go:122] "RemoveContainer" containerID="355991a5e851a775216696e5a196e82a644122d56e0a2dadab25054514cfa466"
Nov 01 20:30:15.945255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:15.945236    6479 scope.go:122] "RemoveContainer" containerID="d6ddfc131c557997af4fb04e8080b42e84098a0f72ad3c205716b26c3844f2e6"
Nov 01 20:30:15.945615 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:15.945580    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m_odh-ai-gateway-infra(9146ed68-c9db-48a0-8263-e884d69c3d10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10"
Nov 01 20:30:16.319218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.319177    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:16.700837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.700804    6479 scope.go:122] "RemoveContainer" containerID="fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985"
Nov 01 20:30:16.956198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.956164    6479 generic.go:356] "Generic (PLEG): container finished" podID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerID="be29a6ca870bab6ade675d50bb46b2dcdf22ea59b376a97239cc6990bd6c795b" exitCode=60
Nov 01 20:30:16.956393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.956196    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" event={"ID":"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba","Type":"ContainerDied","Data":"be29a6ca870bab6ade675d50bb46b2dcdf22ea59b376a97239cc6990bd6c795b"}
Nov 01 20:30:16.956393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.956272    6479 scope.go:122] "RemoveContainer" containerID="4a90c802a63158a5c9450456040886bd2ded10d5184d0027116c42a1634cc9ee"
Nov 01 20:30:16.956759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.956734    6479 scope.go:122] "RemoveContainer" containerID="be29a6ca870bab6ade675d50bb46b2dcdf22ea59b376a97239cc6990bd6c795b"
Nov 01 20:30:16.957225 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:16.957103    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b_odh-ai-gateway-infra(9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba"
Nov 01 20:30:16.976742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:16.976708    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"]
Nov 01 20:30:17.052050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.052004    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:17.181319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.181289    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:17.292070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.292027    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca\") pod \"9146ed68-c9db-48a0-8263-e884d69c3d10\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") "
Nov 01 20:30:17.292284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.292101    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g\" (UniqueName: \"kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g\") pod \"9146ed68-c9db-48a0-8263-e884d69c3d10\" (UID: \"9146ed68-c9db-48a0-8263-e884d69c3d10\") "
Nov 01 20:30:17.292443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.292420    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca" pod "9146ed68-c9db-48a0-8263-e884d69c3d10" (UID: "9146ed68-c9db-48a0-8263-e884d69c3d10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:17.292602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.292586    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9146ed68-c9db-48a0-8263-e884d69c3d10-maas-api-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:17.294353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.294320    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g" pod "9146ed68-c9db-48a0-8263-e884d69c3d10" (UID: "9146ed68-c9db-48a0-8263-e884d69c3d10"). InnerVolumeSpecName "kube-api-access-lfw5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:17.394347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.394301    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfw5g\" (UniqueName: \"kubernetes.io/projected/9146ed68-c9db-48a0-8263-e884d69c3d10-kube-api-access-lfw5g\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:17.835081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.835047    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:17.982095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.982047    6479 generic.go:356] "Generic (PLEG): container finished" podID="cba424b9-8911-4392-81ae-d68379ef9f13" containerID="46d72ec162252e894bf6db5e278597dfb4913ee769aa99dc303bc71637601828" exitCode=60
Nov 01 20:30:17.982095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.982097    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerDied","Data":"46d72ec162252e894bf6db5e278597dfb4913ee769aa99dc303bc71637601828"}
Nov 01 20:30:17.982453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.982159    6479 scope.go:122] "RemoveContainer" containerID="fae56cdd1cdbbe03e0a43cb50ed95dce613314a0055142e719aa828aec80c985"
Nov 01 20:30:17.982693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.982668    6479 scope.go:122] "RemoveContainer" containerID="46d72ec162252e894bf6db5e278597dfb4913ee769aa99dc303bc71637601828"
Nov 01 20:30:17.983160 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:17.983131    6479 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw_odh-ai-gateway-infra(cba424b9-8911-4392-81ae-d68379ef9f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" podUID="cba424b9-8911-4392-81ae-d68379ef9f13"
Nov 01 20:30:17.988233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.988210    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m" event={"ID":"9146ed68-c9db-48a0-8263-e884d69c3d10","Type":"ContainerDied","Data":"a7d10758c76c86706a4596465b836aa858ca97ed081c08949fcdaf48da2cb8e7"}
Nov 01 20:30:17.988438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.988417    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"
Nov 01 20:30:17.988707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:17.988682    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"]
Nov 01 20:30:18.018161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.018100    6479 scope.go:122] "RemoveContainer" containerID="d6ddfc131c557997af4fb04e8080b42e84098a0f72ad3c205716b26c3844f2e6"
Nov 01 20:30:18.055534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.055465    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"]
Nov 01 20:30:18.060342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.060310    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892750-rt99m"]
Nov 01 20:30:18.175973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.175943    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:18.215934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.215820    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca\") pod \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") "
Nov 01 20:30:18.216150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.215981    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf\" (UniqueName: \"kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf\") pod \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\" (UID: \"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba\") "
Nov 01 20:30:18.216320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.216290    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca" pod "9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" (UID: "9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:18.216694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.216671    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-maas-api-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.225679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.219502    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf" pod "9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" (UID: "9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba"). InnerVolumeSpecName "kube-api-access-xt8vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:18.317633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.317546    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt8vf\" (UniqueName: \"kubernetes.io/projected/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba-kube-api-access-xt8vf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.563976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.563933    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:18.708459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.708428    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" path="/var/lib/kubelet/pods/9146ed68-c9db-48a0-8263-e884d69c3d10/volumes"
Nov 01 20:30:18.996939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.996847    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"
Nov 01 20:30:18.996939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.996886    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b" event={"ID":"9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba","Type":"ContainerDied","Data":"114a6abc0b62bc7ecc3340fed2c4370c5cfb995e81a1dfbb377637a71d5aa1cf"}
Nov 01 20:30:18.997412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:18.996960    6479 scope.go:122] "RemoveContainer" containerID="be29a6ca870bab6ade675d50bb46b2dcdf22ea59b376a97239cc6990bd6c795b"
Nov 01 20:30:19.014833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.014797    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"]
Nov 01 20:30:19.017553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.017522    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"]
Nov 01 20:30:19.027500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.027035    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e83b4b-29892750-kgx7b"]
Nov 01 20:30:19.173537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.173501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:19.194862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.194838    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:19.258956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.252161    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca\") pod \"cba424b9-8911-4392-81ae-d68379ef9f13\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") "
Nov 01 20:30:19.258956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.252584    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7\" (UniqueName: \"kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7\") pod \"cba424b9-8911-4392-81ae-d68379ef9f13\" (UID: \"cba424b9-8911-4392-81ae-d68379ef9f13\") "
Nov 01 20:30:19.258956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.253810    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca" pod "cba424b9-8911-4392-81ae-d68379ef9f13" (UID: "cba424b9-8911-4392-81ae-d68379ef9f13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:19.258956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.256521    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7" pod "cba424b9-8911-4392-81ae-d68379ef9f13" (UID: "cba424b9-8911-4392-81ae-d68379ef9f13"). InnerVolumeSpecName "kube-api-access-5twg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:19.354518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.354470    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba424b9-8911-4392-81ae-d68379ef9f13-maas-api-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:19.354518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:19.354508    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5twg7\" (UniqueName: \"kubernetes.io/projected/cba424b9-8911-4392-81ae-d68379ef9f13-kube-api-access-5twg7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.011525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.011488    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"
Nov 01 20:30:20.012459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.012428    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw" event={"ID":"cba424b9-8911-4392-81ae-d68379ef9f13","Type":"ContainerDied","Data":"b2c733af0cd8cf8552297e1a2248f5f6ebd5ed2590e41c9c40929a6503717cb1"}
Nov 01 20:30:20.012606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.012513    6479 scope.go:122] "RemoveContainer" containerID="46d72ec162252e894bf6db5e278597dfb4913ee769aa99dc303bc71637601828"
Nov 01 20:30:20.026452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.026386    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:20.042947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.042917    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"]
Nov 01 20:30:20.049921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.049892    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a6873b-29892750-82lmw"]
Nov 01 20:30:20.532845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.532803    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:30:20.666635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.666593    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:20.727689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.714264    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" path="/var/lib/kubelet/pods/9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba/volumes"
Nov 01 20:30:20.727689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:20.715757    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" path="/var/lib/kubelet/pods/cba424b9-8911-4392-81ae-d68379ef9f13/volumes"
Nov 01 20:30:21.678674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:21.676346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:22.341220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:22.341184    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:22.937939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:22.937905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:23.040673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:23.037775    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:23.532828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:23.532786    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:24.171726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.171682    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:24.728048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728013    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-341a04"]
Nov 01 20:30:24.728593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728579    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:24.728664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728594    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:24.728664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728609    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.728664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728614    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.728664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728628    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.728664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728635    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728695    6479 cpu_manager.go:414] "containerMap: removing container" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728701    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728717    6479 cpu_manager.go:414] "containerMap: removing container" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728722    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728746    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728750    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728835    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728842    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728849    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:24.728854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728856    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:24.729203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728864    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.729203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728872    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:24.729203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.728879    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:24.731984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.731966    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:24.735859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.735830    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\""
Nov 01 20:30:24.738052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.738031    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-341a04"]
Nov 01 20:30:24.844918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.844877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:24.863330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.863292    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjd25\" (UniqueName: \"kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25\") pod \"test-curl-7426-341a04\" (UID: \"ba20db38-32e6-4c99-ad44-07267bbd843f\") " pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:24.965008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.964970    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjd25\" (UniqueName: \"kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25\") pod \"test-curl-7426-341a04\" (UID: \"ba20db38-32e6-4c99-ad44-07267bbd843f\") " pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:24.973226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:24.973189    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjd25\" (UniqueName: \"kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25\") pod \"test-curl-7426-341a04\" (UID: \"ba20db38-32e6-4c99-ad44-07267bbd843f\") " pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:25.055149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:25.055043    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:25.336511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:25.336477    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-341a04"]
Nov 01 20:30:25.339128 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:30:25.339099    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba20db38_32e6_4c99_ad44_07267bbd843f.slice/crio-ef0c3121df2f70bd65657ccb2d0b1a70b58d5a557bfc7bd32558a9f5dec14e70 WatchSource:0}: Error finding container ef0c3121df2f70bd65657ccb2d0b1a70b58d5a557bfc7bd32558a9f5dec14e70: Status 404 returned error can't find the container with id ef0c3121df2f70bd65657ccb2d0b1a70b58d5a557bfc7bd32558a9f5dec14e70
Nov 01 20:30:25.556015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:25.555976    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:25.656497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:25.656401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:25.852308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:25.852268    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:26.066599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.066554    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-341a04" event={"ID":"ba20db38-32e6-4c99-ad44-07267bbd843f","Type":"ContainerStarted","Data":"7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905"}
Nov 01 20:30:26.066832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.066607    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-341a04" event={"ID":"ba20db38-32e6-4c99-ad44-07267bbd843f","Type":"ContainerStarted","Data":"ef0c3121df2f70bd65657ccb2d0b1a70b58d5a557bfc7bd32558a9f5dec14e70"}
Nov 01 20:30:26.081434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.081379    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7426-341a04" podStartSLOduration=2.081361986 podStartE2EDuration="2.081361986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:26.079934466 +0000 UTC m=+1871.495566932" watchObservedRunningTime="2026-11-01 20:30:26.081361986 +0000 UTC m=+1871.496994458"
Nov 01 20:30:26.144565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.144526    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:26.752838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.752790    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-341a04"]
Nov 01 20:30:26.762562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.762520    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-341a04"]
Nov 01 20:30:26.885774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.882061    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:26.906954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.906917    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-ee9a91"]
Nov 01 20:30:26.908708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908679    6479 cpu_manager.go:414] "containerMap: removing container" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" containerName="test-curl-7426-341a04"
Nov 01 20:30:26.908708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908709    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" containerName="test-curl-7426-341a04"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908733    6479 cpu_manager.go:414] "containerMap: removing container" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908739    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908772    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908777    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908792    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908797    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7e7e77-276b-4c5a-8d4c-2914aa42b9ba" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908883    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba424b9-8911-4392-81ae-d68379ef9f13" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908899    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9146ed68-c9db-48a0-8263-e884d69c3d10" containerName="cleanup"
Nov 01 20:30:26.908921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.908922    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" containerName="test-curl-7426-341a04"
Nov 01 20:30:26.912816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.912795    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:26.918822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:26.918795    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-ee9a91"]
Nov 01 20:30:27.093760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:27.093664    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg77q\" (UniqueName: \"kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q\") pod \"test-curl-7426-ee9a91\" (UID: \"165c30dd-50e2-4a11-a761-29fddb64c414\") " pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:27.194885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:27.194851    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg77q\" (UniqueName: \"kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q\") pod \"test-curl-7426-ee9a91\" (UID: \"165c30dd-50e2-4a11-a761-29fddb64c414\") " pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:27.202517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:27.202488    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg77q\" (UniqueName: \"kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q\") pod \"test-curl-7426-ee9a91\" (UID: \"165c30dd-50e2-4a11-a761-29fddb64c414\") " pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:27.236009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:27.235971    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:27.438552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:27.438463    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:27.546984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:27.546953    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-ee9a91"]
Nov 01 20:30:27.549291 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:30:27.549251    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod165c30dd_50e2_4a11_a761_29fddb64c414.slice/crio-00aa98e94b834c824cd27b72499d8f64978b6d52ffb775b1525df37a1bb7ef5b WatchSource:0}: Error finding container 00aa98e94b834c824cd27b72499d8f64978b6d52ffb775b1525df37a1bb7ef5b: Status 404 returned error can't find the container with id 00aa98e94b834c824cd27b72499d8f64978b6d52ffb775b1525df37a1bb7ef5b
Nov 01 20:30:28.058224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.058186    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:28.081638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.081591    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-ee9a91" event={"ID":"165c30dd-50e2-4a11-a761-29fddb64c414","Type":"ContainerStarted","Data":"8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283"}
Nov 01 20:30:28.081874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.081677    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-ee9a91" event={"ID":"165c30dd-50e2-4a11-a761-29fddb64c414","Type":"ContainerStarted","Data":"00aa98e94b834c824cd27b72499d8f64978b6d52ffb775b1525df37a1bb7ef5b"}
Nov 01 20:30:28.081874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.081669    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7426-341a04" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" containerName="test-curl-7426-341a04" containerID="cri-o://7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905" gracePeriod=2
Nov 01 20:30:28.095967 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:28.095931    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-341a04\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:28.096553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.096507    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7426-ee9a91" podStartSLOduration=2.096493243 podStartE2EDuration="2.096493243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:28.093994333 +0000 UTC m=+1873.509626819" watchObservedRunningTime="2026-11-01 20:30:28.096493243 +0000 UTC m=+1873.512125715"
Nov 01 20:30:28.097988 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:28.097956    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-341a04\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:28.664056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.664018    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:28.712447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.712415    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-ee9a91"]
Nov 01 20:30:28.712619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:28.712459    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-ee9a91"]
Nov 01 20:30:29.186438 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:29.186390    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:30:29.546063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:29.546019    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:30.096455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.096417    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7426-ee9a91" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" containerName="test-curl-7426-ee9a91" containerID="cri-o://8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283" gracePeriod=2
Nov 01 20:30:30.098577 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:30.098550    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-ee9a91\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:30.142556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.142497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:30.285070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.285023    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:30.303701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.303674    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:30.306010 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:30.305979    6479 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-ee9a91\" is forbidden: User \"system:node:ip-10-0-133-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.us-east-2.compute.internal' and this object" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:30.339051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.338958    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25\" (UniqueName: \"kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25\") pod \"ba20db38-32e6-4c99-ad44-07267bbd843f\" (UID: \"ba20db38-32e6-4c99-ad44-07267bbd843f\") "
Nov 01 20:30:30.341164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.341128    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25" pod "ba20db38-32e6-4c99-ad44-07267bbd843f" (UID: "ba20db38-32e6-4c99-ad44-07267bbd843f"). InnerVolumeSpecName "kube-api-access-vjd25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:30.440871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.440833    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjd25\" (UniqueName: \"kubernetes.io/projected/ba20db38-32e6-4c99-ad44-07267bbd843f-kube-api-access-vjd25\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:30.709205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.709166    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba20db38-32e6-4c99-ad44-07267bbd843f" path="/var/lib/kubelet/pods/ba20db38-32e6-4c99-ad44-07267bbd843f/volumes"
Nov 01 20:30:30.737982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:30.737947    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:31.106123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.106027    6479 generic.go:356] "Generic (PLEG): container finished" podID="ba20db38-32e6-4c99-ad44-07267bbd843f" containerID="7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905" exitCode=137
Nov 01 20:30:31.106123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.106070    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-341a04"
Nov 01 20:30:31.106123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.106113    6479 scope.go:122] "RemoveContainer" containerID="7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905"
Nov 01 20:30:31.115471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.115452    6479 scope.go:122] "RemoveContainer" containerID="7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905"
Nov 01 20:30:31.115774 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:31.115755    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905\": container with ID starting with 7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905 not found: ID does not exist" containerID="7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905"
Nov 01 20:30:31.115834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.115786    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905"} err="failed to get container status \"7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905\": rpc error: code = NotFound desc = could not find container \"7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905\": container with ID starting with 7b3b51bb086e80eea13a983090ae9304c5236608b664f79507d5c0024a566905 not found: ID does not exist"
Nov 01 20:30:31.335801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.335761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:31.936858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:31.936816    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:32.299057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:32.299029    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:32.370150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:32.370109    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q\" (UniqueName: \"kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q\") pod \"165c30dd-50e2-4a11-a761-29fddb64c414\" (UID: \"165c30dd-50e2-4a11-a761-29fddb64c414\") "
Nov 01 20:30:32.372386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:32.372358    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q" pod "165c30dd-50e2-4a11-a761-29fddb64c414" (UID: "165c30dd-50e2-4a11-a761-29fddb64c414"). InnerVolumeSpecName "kube-api-access-zg77q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:32.472336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:32.472239    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg77q\" (UniqueName: \"kubernetes.io/projected/165c30dd-50e2-4a11-a761-29fddb64c414-kube-api-access-zg77q\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:32.567536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:32.567493    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:32.710463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:32.710423    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" path="/var/lib/kubelet/pods/165c30dd-50e2-4a11-a761-29fddb64c414/volumes"
Nov 01 20:30:33.120995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.120956    6479 generic.go:356] "Generic (PLEG): container finished" podID="165c30dd-50e2-4a11-a761-29fddb64c414" containerID="8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283" exitCode=137
Nov 01 20:30:33.121250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.120999    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-ee9a91"
Nov 01 20:30:33.121250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.121064    6479 scope.go:122] "RemoveContainer" containerID="8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283"
Nov 01 20:30:33.130047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.130024    6479 scope.go:122] "RemoveContainer" containerID="8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283"
Nov 01 20:30:33.130359 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:33.130337    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283\": container with ID starting with 8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283 not found: ID does not exist" containerID="8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283"
Nov 01 20:30:33.130415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.130368    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283"} err="failed to get container status \"8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283\": rpc error: code = NotFound desc = could not find container \"8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283\": container with ID starting with 8edab5dd0475dfce6ef6b6fcad6aa5a2ae76363a9a586acf42c7510bfed59283 not found: ID does not exist"
Nov 01 20:30:33.259051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.259009    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:33.877422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:33.877372    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:34.467140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:34.467080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:35.155308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:35.155273    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:35.586564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:35.586526    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"]
Nov 01 20:30:35.602620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:35.602583    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"]
Nov 01 20:30:35.652473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:35.652438    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:35.752072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:35.752029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:36.178833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:36.178795    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:30:36.214713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:36.214680    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:37.271420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:37.271381    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:37.375094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:37.375056    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:37.924083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:37.924042    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:38.404664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:38.404615    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:39.364320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:39.364286    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:40.178467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:40.178431    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:40.587275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:40.587217    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq" podUID="e20d8956-a771-4355-ac23-c47ff178585b" containerName="payload-processing" containerID="cri-o://011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617" gracePeriod=25
Nov 01 20:30:40.603823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:40.603781    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96" podUID="6b63897e-3823-4d01-a306-c9e921bb82fa" containerName="payload-pre-processing" containerID="cri-o://d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4" gracePeriod=25
Nov 01 20:30:40.886726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:40.886689    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:30:40.924287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:40.924255    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:30:41.014454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.014413    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g\" (UniqueName: \"kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g\") pod \"6b63897e-3823-4d01-a306-c9e921bb82fa\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") "
Nov 01 20:30:41.014697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.014525    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5\" (UniqueName: \"kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5\") pod \"e20d8956-a771-4355-ac23-c47ff178585b\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") "
Nov 01 20:30:41.014697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.014598    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume\") pod \"e20d8956-a771-4355-ac23-c47ff178585b\" (UID: \"e20d8956-a771-4355-ac23-c47ff178585b\") "
Nov 01 20:30:41.014841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.014742    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume\") pod \"6b63897e-3823-4d01-a306-c9e921bb82fa\" (UID: \"6b63897e-3823-4d01-a306-c9e921bb82fa\") "
Nov 01 20:30:41.015103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.015076    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume" pod "e20d8956-a771-4355-ac23-c47ff178585b" (UID: "e20d8956-a771-4355-ac23-c47ff178585b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:41.015240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.015215    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume" pod "6b63897e-3823-4d01-a306-c9e921bb82fa" (UID: "6b63897e-3823-4d01-a306-c9e921bb82fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:41.015321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.015288    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20d8956-a771-4355-ac23-c47ff178585b-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:41.017045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.017016    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g" pod "6b63897e-3823-4d01-a306-c9e921bb82fa" (UID: "6b63897e-3823-4d01-a306-c9e921bb82fa"). InnerVolumeSpecName "kube-api-access-tdd9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:41.017154 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.017087    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5" pod "e20d8956-a771-4355-ac23-c47ff178585b" (UID: "e20d8956-a771-4355-ac23-c47ff178585b"). InnerVolumeSpecName "kube-api-access-69pn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:41.071499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.071463    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:41.116597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.116504    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63897e-3823-4d01-a306-c9e921bb82fa-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:41.116597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.116549    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdd9g\" (UniqueName: \"kubernetes.io/projected/6b63897e-3823-4d01-a306-c9e921bb82fa-kube-api-access-tdd9g\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:41.116597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.116567    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69pn5\" (UniqueName: \"kubernetes.io/projected/e20d8956-a771-4355-ac23-c47ff178585b-kube-api-access-69pn5\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:41.215038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.214997    6479 generic.go:356] "Generic (PLEG): container finished" podID="6b63897e-3823-4d01-a306-c9e921bb82fa" containerID="d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4" exitCode=0
Nov 01 20:30:41.215534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.215143    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96" event={"ID":"6b63897e-3823-4d01-a306-c9e921bb82fa","Type":"ContainerDied","Data":"d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4"}
Nov 01 20:30:41.215534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.215195    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96" event={"ID":"6b63897e-3823-4d01-a306-c9e921bb82fa","Type":"ContainerDied","Data":"8aceb41e8735f036976138038802797021dda365f0f8b97545c49272d4b56928"}
Nov 01 20:30:41.215534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.215222    6479 scope.go:122] "RemoveContainer" containerID="d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4"
Nov 01 20:30:41.215534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.215382    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"
Nov 01 20:30:41.219580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.219546    6479 generic.go:356] "Generic (PLEG): container finished" podID="e20d8956-a771-4355-ac23-c47ff178585b" containerID="011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617" exitCode=0
Nov 01 20:30:41.219772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.219698    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq" event={"ID":"e20d8956-a771-4355-ac23-c47ff178585b","Type":"ContainerDied","Data":"011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617"}
Nov 01 20:30:41.219772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.219745    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq" event={"ID":"e20d8956-a771-4355-ac23-c47ff178585b","Type":"ContainerDied","Data":"41bd2f4d03abbe9a825fb0e5f0ce6f13e623cdd6b522a101524023e8749e69b2"}
Nov 01 20:30:41.219898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.219815    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"
Nov 01 20:30:41.229223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.229198    6479 scope.go:122] "RemoveContainer" containerID="d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4"
Nov 01 20:30:41.229914 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:41.229694    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4\": container with ID starting with d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4 not found: ID does not exist" containerID="d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4"
Nov 01 20:30:41.229914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.229729    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4"} err="failed to get container status \"d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4\": rpc error: code = NotFound desc = could not find container \"d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4\": container with ID starting with d9737bbeedb92fe296f8fb8ad891ab547415ee35da7f54cad721e8280be7f5a4 not found: ID does not exist"
Nov 01 20:30:41.229914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.229750    6479 scope.go:122] "RemoveContainer" containerID="011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617"
Nov 01 20:30:41.239195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.239174    6479 scope.go:122] "RemoveContainer" containerID="011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617"
Nov 01 20:30:41.239480 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:41.239455    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617\": container with ID starting with 011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617 not found: ID does not exist" containerID="011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617"
Nov 01 20:30:41.239546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.239491    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617"} err="failed to get container status \"011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617\": rpc error: code = NotFound desc = could not find container \"011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617\": container with ID starting with 011e635625bbcd53419f35c34a97184f0dfecf927eaa04bb84a0634495f60617 not found: ID does not exist"
Nov 01 20:30:41.247372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.247335    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"]
Nov 01 20:30:41.249073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.249050    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d2dc0ad5-96c44856c-pbt96"]
Nov 01 20:30:41.259057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.259026    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"]
Nov 01 20:30:41.261361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.261339    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2dc0ad5-6fbb47f585-wtmzq"]
Nov 01 20:30:41.757980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:41.757917    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:42.533275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:42.533236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:42.708662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:42.708608    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b63897e-3823-4d01-a306-c9e921bb82fa" path="/var/lib/kubelet/pods/6b63897e-3823-4d01-a306-c9e921bb82fa/volumes"
Nov 01 20:30:42.709112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:42.709097    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20d8956-a771-4355-ac23-c47ff178585b" path="/var/lib/kubelet/pods/e20d8956-a771-4355-ac23-c47ff178585b/volumes"
Nov 01 20:30:42.849898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:42.849815    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"]
Nov 01 20:30:42.850075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:42.850039    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy" containerID="cri-o://b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe" gracePeriod=30
Nov 01 20:30:43.110813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:43.110731    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"]
Nov 01 20:30:43.110973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:43.110951    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy" containerID="cri-o://c4e12420e8c2f3e01f3afaf6f53ba919c11d7a526f153da11ac522b1138df298" gracePeriod=30
Nov 01 20:30:43.193107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:43.193066    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:43.268132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:43.268101    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:43.707557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:43.707521    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:44.043849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:44.043753    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:44.349767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:44.349615    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:44.478314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:44.478270    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:44.584764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:44.584720    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:44.840058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:44.840017    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:45.141278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:45.141201    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:45.952679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:45.952611    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:46.651543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:46.651503    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:46.886920 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:46.886887    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:47.256907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:47.256869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:47.847692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:47.847661    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:48.139906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.139875    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:30:48.245302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245268    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245332    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245359    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245406    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245494    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245544    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245708    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245726    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.245764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245744    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.245764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245737    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:48.246045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245879    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy\") pod \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\" (UID: \"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8\") "
Nov 01 20:30:48.246045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.245917    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.246155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246113    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.246198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246148    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.246623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246597    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.246746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246628    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.246746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246676    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.246746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246693    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.246746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.246708    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.248046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.248019    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "kube-api-access-9l52v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:48.248204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.248187    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:48.248481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.248459    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.248795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.248779    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo" pod "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" (UID: "9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:30:48.286863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.286832    6479 generic.go:356] "Generic (PLEG): container finished" podID="d0f206db-9d47-4687-9cdc-884371b94797" containerID="c4e12420e8c2f3e01f3afaf6f53ba919c11d7a526f153da11ac522b1138df298" exitCode=0
Nov 01 20:30:48.287319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.286906    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" event={"ID":"d0f206db-9d47-4687-9cdc-884371b94797","Type":"ContainerDied","Data":"c4e12420e8c2f3e01f3afaf6f53ba919c11d7a526f153da11ac522b1138df298"}
Nov 01 20:30:48.289107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.289083    6479 generic.go:356] "Generic (PLEG): container finished" podID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerID="b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe" exitCode=0
Nov 01 20:30:48.289225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.289144    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"
Nov 01 20:30:48.289225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.289162    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" event={"ID":"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8","Type":"ContainerDied","Data":"b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe"}
Nov 01 20:30:48.289225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.289206    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs" event={"ID":"9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8","Type":"ContainerDied","Data":"ba2f92f51815869682974f22ad8481d7d2b79c6f3a637a83f82a08f25cad087b"}
Nov 01 20:30:48.289387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.289238    6479 scope.go:122] "RemoveContainer" containerID="b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe"
Nov 01 20:30:48.305501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.305471    6479 scope.go:122] "RemoveContainer" containerID="7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"
Nov 01 20:30:48.318026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.317970    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"]
Nov 01 20:30:48.322664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.322613    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-7f2bs"]
Nov 01 20:30:48.323958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.323937    6479 scope.go:122] "RemoveContainer" containerID="b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe"
Nov 01 20:30:48.325018 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:48.324994    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe\": container with ID starting with b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe not found: ID does not exist" containerID="b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe"
Nov 01 20:30:48.325141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.325033    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe"} err="failed to get container status \"b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe\": rpc error: code = NotFound desc = could not find container \"b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe\": container with ID starting with b62a1f38f6652a9ca02ff9247c51c87c47b278c14120e534e43b57243297f7fe not found: ID does not exist"
Nov 01 20:30:48.325141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.325058    6479 scope.go:122] "RemoveContainer" containerID="7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"
Nov 01 20:30:48.325577 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:48.325535    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470\": container with ID starting with 7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470 not found: ID does not exist" containerID="7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"
Nov 01 20:30:48.325577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.325576    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470"} err="failed to get container status \"7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470\": rpc error: code = NotFound desc = could not find container \"7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470\": container with ID starting with 7bff9c9a6846981920b63ec5e4ea9a692ee29ce28671d7e21f74fa241cc3d470 not found: ID does not exist"
Nov 01 20:30:48.349687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.347987    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.349687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.348021    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.349687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.348037    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l52v\" (UniqueName: \"kubernetes.io/projected/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-kube-api-access-9l52v\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.349687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.348052    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.366846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.366771    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:30:48.463822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.463784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.565710    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.565774    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.565935    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.565964    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.566003    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.566030    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.566067    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.566099    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.566158    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token\") pod \"d0f206db-9d47-4687-9cdc-884371b94797\" (UID: \"d0f206db-9d47-4687-9cdc-884371b94797\") "
Nov 01 20:30:48.567831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.567401    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.568532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.567894    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.568532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.568452    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.568532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.568509    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.569043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.569007    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:48.573150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.572891    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:30:48.573349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.573320    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "kube-api-access-h6t6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:48.574604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.574573    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:48.575106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.575085    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy" pod "d0f206db-9d47-4687-9cdc-884371b94797" (UID: "d0f206db-9d47-4687-9cdc-884371b94797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:48.667883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667795    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f206db-9d47-4687-9cdc-884371b94797-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.667883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667833    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.667883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667849    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.667883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667867    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f206db-9d47-4687-9cdc-884371b94797-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.667883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667881    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.668177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667897    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.668177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667911    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.668177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667925    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6t6n\" (UniqueName: \"kubernetes.io/projected/d0f206db-9d47-4687-9cdc-884371b94797-kube-api-access-h6t6n\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.668177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.667941    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f206db-9d47-4687-9cdc-884371b94797-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:48.708982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:48.708947    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" path="/var/lib/kubelet/pods/9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8/volumes"
Nov 01 20:30:49.044298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.044263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:49.302543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.302459    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"
Nov 01 20:30:49.302543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.302470    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5" event={"ID":"d0f206db-9d47-4687-9cdc-884371b94797","Type":"ContainerDied","Data":"11c3a40edac8ea93fe618243b1f78a6566feba12cc3b779f3a3cf6735e1ff5e3"}
Nov 01 20:30:49.302543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.302539    6479 scope.go:122] "RemoveContainer" containerID="c4e12420e8c2f3e01f3afaf6f53ba919c11d7a526f153da11ac522b1138df298"
Nov 01 20:30:49.320695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.320627    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"]
Nov 01 20:30:49.324388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.324360    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-2j8w5"]
Nov 01 20:30:49.646820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:49.646731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:50.180430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:50.180390    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:50.356574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:50.356536    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:50.708715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:50.708677    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f206db-9d47-4687-9cdc-884371b94797" path="/var/lib/kubelet/pods/d0f206db-9d47-4687-9cdc-884371b94797/volumes"
Nov 01 20:30:51.034155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:51.034068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:51.636348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:51.636303    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:51.743062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:51.743026    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:51.854628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:51.854582    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:51.943678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:51.943556    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:52.156140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:52.156102    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:52.334498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:52.334461    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:52.952779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:52.952744    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:30:52.955337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:52.955309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:53.500886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:53.500848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:53.601144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:53.601060    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:54.247197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:54.247164    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:54.856223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:54.856188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:55.464011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:55.463970    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:56.055505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:56.055466    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:56.663815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:56.663771    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:57.316348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:57.316041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:57.791288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:57.791249    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:57.888760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:57.888722    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:57.965758 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:57.965710    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:58.742233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:58.742196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:59.172345 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:30:59.172259    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:30:59.335132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:59.335095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:30:59.921181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:30:59.921149    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:00.701347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:00.701310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:01.435243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:01.435206    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:02.248138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:02.248093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:03.128073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:03.128008    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:03.952441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:03.952401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:04.573320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:04.573281    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:05.270847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:05.270804    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:05.869274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:05.869225    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:06.640434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:06.640396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:07.150078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:07.150036    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:07.523088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:07.523054    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:31:07.842906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:07.842825    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:08.649381 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:08.649343    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:09.460657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:09.460607    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:10.245837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:10.245775    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:10.945855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:10.945821    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:11.186544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:11.186503    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:11.583165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:11.583122    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:12.369632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:12.369587    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:12.793136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:12.793079    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:13.156953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:13.156875    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:13.259171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:13.259132    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:13.768320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:13.768288    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:14.438087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:14.438050    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:15.044707 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:15.044669    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:15.729693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:15.729632    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:16.434948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:16.434916    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:17.146333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:17.146293    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:17.758796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:17.758760    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:18.363478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:18.363442    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:18.567701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:18.566788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:19.257605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:19.257553    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:19.944837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:19.944804    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:20.548695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:20.548655    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:21.144958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:21.144923    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:21.743795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:21.743738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:22.347425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:22.347376    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:22.948567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:22.948529    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:23.039792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:23.039754    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:23.549211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:23.549175    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:24.166615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:24.166575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:24.759733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:24.759688    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:25.471467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:25.471428    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:26.059508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:26.059473    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:26.633695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:26.633632    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:27.261836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:27.261794    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:27.868100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:27.868063    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:27.957601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:27.956679    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:31:28.479773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:28.479677    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:29.031451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:29.031413    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:29.171983 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:31:29.171942    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:31:29.376760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:29.376641    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:29.747729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:29.747695    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:30.341987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:30.341948    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:30.968633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:30.968593    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:31.565408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:31.565373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:32.148735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:32.148699    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:32.759473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:32.759437    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:33.376750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:33.376704    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:34.036796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:34.036763    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:34.341963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:34.341859    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"]
Nov 01 20:31:34.367755 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:34.367724    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"]
Nov 01 20:31:34.436662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:34.436618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:34.764456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:34.764415    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:34.880305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:34.880270    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:35.428373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:35.428303    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:36.163105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:36.163052    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:36.798435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:36.798398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:37.477934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:37.477883    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:38.058247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:38.058210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:38.658987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:38.658948    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:39.243817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:39.243784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:39.654281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:39.654188    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:31:39.836450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:39.836407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:40.044704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:40.044658    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:40.140598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:40.140557    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:40.760884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:40.760841    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:41.559014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:41.558975    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:41.687834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:41.687803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:42.152265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:42.152229    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:42.739616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:42.739556    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:44.257112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:44.257068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:44.742857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:44.742824    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:45.363377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:45.363322    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:46.097257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:46.097217    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:46.960164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:46.960127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:47.873441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:47.873401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:48.656392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:48.656351    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:48.810969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:48.810929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:49.364829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.364777    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main" containerID="cri-o://bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f" gracePeriod=45
Nov 01 20:31:49.389280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.389243    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main" containerID="cri-o://0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab" gracePeriod=45
Nov 01 20:31:49.409549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.409516    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"]
Nov 01 20:31:49.409772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.409750    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy" containerID="cri-o://1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a" gracePeriod=30
Nov 01 20:31:49.440033 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.439765    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:49.691350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.690921    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:31:49.698287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.698248    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:31:49.823397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.823352    6479 generic.go:356] "Generic (PLEG): container finished" podID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerID="0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab" exitCode=0
Nov 01 20:31:49.823657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.823607    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" event={"ID":"e2c852ae-b370-4114-bcb1-1c5fd54e8328","Type":"ContainerDied","Data":"0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab"}
Nov 01 20:31:49.823732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.823713    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6" event={"ID":"e2c852ae-b370-4114-bcb1-1c5fd54e8328","Type":"ContainerDied","Data":"420e7eb226ad618db958534c4c90df9fe1e18820fa83c022ab1a3005b310955b"}
Nov 01 20:31:49.823802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.823745    6479 scope.go:122] "RemoveContainer" containerID="0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab"
Nov 01 20:31:49.823965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.823950    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"
Nov 01 20:31:49.827018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.826988    6479 generic.go:356] "Generic (PLEG): container finished" podID="f54274d3-3b0d-434f-8da7-6457847570fc" containerID="bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f" exitCode=0
Nov 01 20:31:49.827219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.827096    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" event={"ID":"f54274d3-3b0d-434f-8da7-6457847570fc","Type":"ContainerDied","Data":"bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f"}
Nov 01 20:31:49.827219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.827126    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2" event={"ID":"f54274d3-3b0d-434f-8da7-6457847570fc","Type":"ContainerDied","Data":"afa7f1cc2eab98eb9fc357262b8ea17580859bb96b17a690306a05bb25be7735"}
Nov 01 20:31:49.827219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.827182    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"
Nov 01 20:31:49.827410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.827369    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs\") pod \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") "
Nov 01 20:31:49.827498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.827476    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache\") pod \"f54274d3-3b0d-434f-8da7-6457847570fc\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") "
Nov 01 20:31:49.827842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.827817    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir" pod "e2c852ae-b370-4114-bcb1-1c5fd54e8328" (UID: "e2c852ae-b370-4114-bcb1-1c5fd54e8328"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.828034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.828012    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache" pod "f54274d3-3b0d-434f-8da7-6457847570fc" (UID: "f54274d3-3b0d-434f-8da7-6457847570fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.828106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.828049    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir\") pod \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") "
Nov 01 20:31:49.828179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.828162    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm\") pod \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") "
Nov 01 20:31:49.828612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.828522    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm\") pod \"f54274d3-3b0d-434f-8da7-6457847570fc\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") "
Nov 01 20:31:49.828949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.828929    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs\") pod \"f54274d3-3b0d-434f-8da7-6457847570fc\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") "
Nov 01 20:31:49.829304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829282    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home\") pod \"f54274d3-3b0d-434f-8da7-6457847570fc\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") "
Nov 01 20:31:49.829407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829326    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache\") pod \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") "
Nov 01 20:31:49.829474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829453    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr\" (UniqueName: \"kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr\") pod \"f54274d3-3b0d-434f-8da7-6457847570fc\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") "
Nov 01 20:31:49.829627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829606    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76\" (UniqueName: \"kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76\") pod \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") "
Nov 01 20:31:49.829748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829694    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir\") pod \"f54274d3-3b0d-434f-8da7-6457847570fc\" (UID: \"f54274d3-3b0d-434f-8da7-6457847570fc\") "
Nov 01 20:31:49.829748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829726    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home\") pod \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\" (UID: \"e2c852ae-b370-4114-bcb1-1c5fd54e8328\") "
Nov 01 20:31:49.829748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.829723    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache" pod "e2c852ae-b370-4114-bcb1-1c5fd54e8328" (UID: "e2c852ae-b370-4114-bcb1-1c5fd54e8328"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.830169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830032    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home" pod "f54274d3-3b0d-434f-8da7-6457847570fc" (UID: "f54274d3-3b0d-434f-8da7-6457847570fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.830425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830403    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.830509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830463    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir" pod "f54274d3-3b0d-434f-8da7-6457847570fc" (UID: "f54274d3-3b0d-434f-8da7-6457847570fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.830509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830475    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home" pod "e2c852ae-b370-4114-bcb1-1c5fd54e8328" (UID: "e2c852ae-b370-4114-bcb1-1c5fd54e8328"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.830641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830539    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.830641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830632    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.830777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.830666    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.831323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.831296    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs" pod "e2c852ae-b370-4114-bcb1-1c5fd54e8328" (UID: "e2c852ae-b370-4114-bcb1-1c5fd54e8328"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:49.834488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.833203    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm" pod "f54274d3-3b0d-434f-8da7-6457847570fc" (UID: "f54274d3-3b0d-434f-8da7-6457847570fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.834488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.833219    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs" pod "f54274d3-3b0d-434f-8da7-6457847570fc" (UID: "f54274d3-3b0d-434f-8da7-6457847570fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:49.834488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.833381    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm" pod "e2c852ae-b370-4114-bcb1-1c5fd54e8328" (UID: "e2c852ae-b370-4114-bcb1-1c5fd54e8328"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.834488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.834429    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76" pod "e2c852ae-b370-4114-bcb1-1c5fd54e8328" (UID: "e2c852ae-b370-4114-bcb1-1c5fd54e8328"). InnerVolumeSpecName "kube-api-access-2tg76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:49.835028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.835004    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr" pod "f54274d3-3b0d-434f-8da7-6457847570fc" (UID: "f54274d3-3b0d-434f-8da7-6457847570fc"). InnerVolumeSpecName "kube-api-access-74zfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:49.859356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.859315    6479 scope.go:122] "RemoveContainer" containerID="0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab"
Nov 01 20:31:49.859798 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:31:49.859776    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab\": container with ID starting with 0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab not found: ID does not exist" containerID="0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab"
Nov 01 20:31:49.859883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.859806    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab"} err="failed to get container status \"0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab\": rpc error: code = NotFound desc = could not find container \"0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab\": container with ID starting with 0530d906c1e9c1c0e03662ed49781f0e1acabd1dca47d647bbc92368157193ab not found: ID does not exist"
Nov 01 20:31:49.859883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.859826    6479 scope.go:122] "RemoveContainer" containerID="bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f"
Nov 01 20:31:49.869660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.869622    6479 scope.go:122] "RemoveContainer" containerID="bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f"
Nov 01 20:31:49.869974 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:31:49.869945    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f\": container with ID starting with bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f not found: ID does not exist" containerID="bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f"
Nov 01 20:31:49.870042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.869984    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f"} err="failed to get container status \"bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f\": rpc error: code = NotFound desc = could not find container \"bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f\": container with ID starting with bd35ab992885e1ddaa4c5e706b273c3e545d84e90f84669a4a9676651121055f not found: ID does not exist"
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.931981    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932012    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54274d3-3b0d-434f-8da7-6457847570fc-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932022    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zfr\" (UniqueName: \"kubernetes.io/projected/f54274d3-3b0d-434f-8da7-6457847570fc-kube-api-access-74zfr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932031    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tg76\" (UniqueName: \"kubernetes.io/projected/e2c852ae-b370-4114-bcb1-1c5fd54e8328-kube-api-access-2tg76\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932040    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54274d3-3b0d-434f-8da7-6457847570fc-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932049    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932056    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c852ae-b370-4114-bcb1-1c5fd54e8328-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.932064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:49.932063    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c852ae-b370-4114-bcb1-1c5fd54e8328-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.091281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.091242    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:50.151372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.151343    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"]
Nov 01 20:31:50.153686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.153662    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/simulator-w0-3a6480-kserve-596895b97-ggct6"]
Nov 01 20:31:50.162467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.162443    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"]
Nov 01 20:31:50.167312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.167285    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-2-w0-3a6480-kserve-86bdf5ffb8-zfwc2"]
Nov 01 20:31:50.665053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.665017    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:50.724736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.724703    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" path="/var/lib/kubelet/pods/e2c852ae-b370-4114-bcb1-1c5fd54e8328/volumes"
Nov 01 20:31:50.725222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:50.725204    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" path="/var/lib/kubelet/pods/f54274d3-3b0d-434f-8da7-6457847570fc/volumes"
Nov 01 20:31:51.239930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:51.239895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:51.878693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:51.878636    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:52.560331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:52.560291    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:53.174844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:53.174807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:53.761963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:53.761922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:54.450537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.450504    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:54.551417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.551384    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:54.694991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.694960    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:31:54.746136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.746102    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:31:54.789779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.789742    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.789951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.789788    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.789951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.789844    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.789951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.789913    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.790128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.789955    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.790128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790047    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.790128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790071    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:54.790128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790084    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.790128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790120    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.790399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790144    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket\") pod \"9af32888-a7a6-444e-be15-22035c3373ac\" (UID: \"9af32888-a7a6-444e-be15-22035c3373ac\") "
Nov 01 20:31:54.790463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790392    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:54.790463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790411    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:54.790463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790407    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:54.790896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790872    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:54.791068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.790924    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.791204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.791188    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af32888-a7a6-444e-be15-22035c3373ac-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.791315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.791300    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.791417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.791404    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.792744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.792712    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:54.793003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.792979    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:54.793090 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.793014    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "kube-api-access-4m9sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:54.793163 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.793148    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo" pod "9af32888-a7a6-444e-be15-22035c3373ac" (UID: "9af32888-a7a6-444e-be15-22035c3373ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:31:54.870522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.870485    6479 generic.go:356] "Generic (PLEG): container finished" podID="9af32888-a7a6-444e-be15-22035c3373ac" containerID="1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a" exitCode=0
Nov 01 20:31:54.870767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.870532    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"
Nov 01 20:31:54.870767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.870559    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" event={"ID":"9af32888-a7a6-444e-be15-22035c3373ac","Type":"ContainerDied","Data":"1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a"}
Nov 01 20:31:54.870767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.870616    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc" event={"ID":"9af32888-a7a6-444e-be15-22035c3373ac","Type":"ContainerDied","Data":"b16fc3371159af41faa1f4fe5026c446297c896d51de108caf8002bb1abfc106"}
Nov 01 20:31:54.870767 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.870638    6479 scope.go:122] "RemoveContainer" containerID="1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a"
Nov 01 20:31:54.882013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.881982    6479 scope.go:122] "RemoveContainer" containerID="ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"
Nov 01 20:31:54.892232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.892204    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af32888-a7a6-444e-be15-22035c3373ac-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.892351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.892231    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9sv\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-kube-api-access-4m9sv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.892351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.892248    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.892351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.892258    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af32888-a7a6-444e-be15-22035c3373ac-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.892351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.892268    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af32888-a7a6-444e-be15-22035c3373ac-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:54.893738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.893717    6479 scope.go:122] "RemoveContainer" containerID="1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a"
Nov 01 20:31:54.894076 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:31:54.894055    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a\": container with ID starting with 1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a not found: ID does not exist" containerID="1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a"
Nov 01 20:31:54.894455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.894088    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a"} err="failed to get container status \"1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a\": rpc error: code = NotFound desc = could not find container \"1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a\": container with ID starting with 1f60e59c59fbd636be87a1ab2677fef4419e6911e6214c57dc55977655eebd2a not found: ID does not exist"
Nov 01 20:31:54.894455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.894143    6479 scope.go:122] "RemoveContainer" containerID="ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"
Nov 01 20:31:54.894592 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:31:54.894489    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64\": container with ID starting with ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64 not found: ID does not exist" containerID="ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"
Nov 01 20:31:54.894592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.894521    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64"} err="failed to get container status \"ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64\": rpc error: code = NotFound desc = could not find container \"ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64\": container with ID starting with ae0ae7365d6d8009e8991fc0eec86857d3e4cd4410601f8ae32557e46348ac64 not found: ID does not exist"
Nov 01 20:31:54.896313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.896292    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"]
Nov 01 20:31:54.899555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:54.899532    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-7rmrc"]
Nov 01 20:31:55.161550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:55.161514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:55.518454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:55.518421    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"]
Nov 01 20:31:55.533910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:55.533880    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"]
Nov 01 20:31:55.605020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:55.604899    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:55.686613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:55.686577    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:55.788692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:55.788578    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:56.200212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:56.200175    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:56.708529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:56.708496    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af32888-a7a6-444e-be15-22035c3373ac" path="/var/lib/kubelet/pods/9af32888-a7a6-444e-be15-22035c3373ac/volumes"
Nov 01 20:31:56.749706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:56.749667    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:57.335845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:57.335809    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:57.980664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:57.980572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:58.783730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:58.783681    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:31:59.171527 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:31:59.171421    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:31:59.498457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:31:59.498383    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:00.157323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.157285    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:00.519774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.519730    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerName="payload-processing" containerID="cri-o://0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32" gracePeriod=25
Nov 01 20:32:00.534231 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.534195    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerName="payload-pre-processing" containerID="cri-o://5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570" gracePeriod=25
Nov 01 20:32:00.664130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.664080    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"]
Nov 01 20:32:00.664500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.664463    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" podUID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" containerName="maas-api" containerID="cri-o://2cc78417496074896773678a933f83b2d6b7b85330f282eff6d4f4f0833c31d2" gracePeriod=30
Nov 01 20:32:00.686468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.686401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:00.688562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.688528    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" podUID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.33:8443/health\": dial tcp 10.134.0.33:8443: connect: connection refused"
Nov 01 20:32:00.865926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.865888    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:00.911400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.911363    6479 generic.go:356] "Generic (PLEG): container finished" podID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" containerID="2cc78417496074896773678a933f83b2d6b7b85330f282eff6d4f4f0833c31d2" exitCode=0
Nov 01 20:32:00.911400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.911410    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" event={"ID":"824bbcf5-e5e1-45a4-958a-bedb8740ad3e","Type":"ContainerDied","Data":"2cc78417496074896773678a933f83b2d6b7b85330f282eff6d4f4f0833c31d2"}
Nov 01 20:32:00.957746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:00.957714    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:32:01.077011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.076919    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k\" (UniqueName: \"kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k\") pod \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") "
Nov 01 20:32:01.077011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.076980    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca\") pod \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") "
Nov 01 20:32:01.077257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.077011    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls\") pod \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") "
Nov 01 20:32:01.077257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.077073    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs\") pod \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\" (UID: \"824bbcf5-e5e1-45a4-958a-bedb8740ad3e\") "
Nov 01 20:32:01.077499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.077464    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca" pod "824bbcf5-e5e1-45a4-958a-bedb8740ad3e" (UID: "824bbcf5-e5e1-45a4-958a-bedb8740ad3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:01.077697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.077680    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:01.079466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.079438    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k" pod "824bbcf5-e5e1-45a4-958a-bedb8740ad3e" (UID: "824bbcf5-e5e1-45a4-958a-bedb8740ad3e"). InnerVolumeSpecName "kube-api-access-vsf8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:01.079699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.079673    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs" pod "824bbcf5-e5e1-45a4-958a-bedb8740ad3e" (UID: "824bbcf5-e5e1-45a4-958a-bedb8740ad3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:01.079814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.079695    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls" pod "824bbcf5-e5e1-45a4-958a-bedb8740ad3e" (UID: "824bbcf5-e5e1-45a4-958a-bedb8740ad3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:01.179529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.179494    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:01.179529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.179530    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsf8k\" (UniqueName: \"kubernetes.io/projected/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-kube-api-access-vsf8k\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:01.179769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.179548    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824bbcf5-e5e1-45a4-958a-bedb8740ad3e-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:01.382694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.382562    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:01.557688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.557610    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:01.645640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.645550    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:01.918979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.918896    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"
Nov 01 20:32:01.918979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.918929    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd" event={"ID":"824bbcf5-e5e1-45a4-958a-bedb8740ad3e","Type":"ContainerDied","Data":"b825dcaa69824267c7e49270c79dd355cbd8322843d81f4d8da0e12b5fde5907"}
Nov 01 20:32:01.919172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.918997    6479 scope.go:122] "RemoveContainer" containerID="2cc78417496074896773678a933f83b2d6b7b85330f282eff6d4f4f0833c31d2"
Nov 01 20:32:01.941291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.941263    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"]
Nov 01 20:32:01.945983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:01.945958    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3a6480-5445f6f8bb-lxgxd"]
Nov 01 20:32:02.200609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:02.200576    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:02.711290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:02.711255    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" path="/var/lib/kubelet/pods/824bbcf5-e5e1-45a4-958a-bedb8740ad3e/volumes"
Nov 01 20:32:02.866488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:02.866456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:03.651190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:03.651155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:04.238203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:04.238155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:04.836944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:04.836901    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:05.508111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:05.508073    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:06.247663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:06.247618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:06.860304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:06.860242    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:07.454220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:07.454176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:07.732791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:07.732706    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:07.814666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:07.814610    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:07.814843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:07.814700    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:07.832389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:07.832355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:08.161348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:08.161258    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:08.257801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:08.257763    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:08.593030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:08.592993    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"]
Nov 01 20:32:08.598779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:08.598737    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" containerID="cri-o://1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829" gracePeriod=30
Nov 01 20:32:08.644960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:08.644908    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:08.854874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:08.854783    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:09.013675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:09.013611    6479 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:32:09.013865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:09.013718    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:32:09.365039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:09.365004    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:09.582995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:09.582955    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:10.182953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:10.182915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:10.283196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:10.283161    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:10.899159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:10.899119    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:11.511232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:11.511201    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:11.687317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:11.687278    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:11.791491 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:11.791413    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:11.895535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:11.895501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:11.991584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:11.991544    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:12.249945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:12.249905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:12.818418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:12.818356    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:12.896658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:12.896624    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:13.095107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.094983    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"]
Nov 01 20:32:13.095629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.095554    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy" containerID="cri-o://a132552396c0291abae65c540214b828b405786aa338494db644f46cfebd18ba" gracePeriod=30
Nov 01 20:32:13.141695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.141624    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:13.241586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.241522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:13.873851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.873767    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:32:13.934602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.934566    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:13.934795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.934627    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:13.934795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.934680    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:13.934795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.934735    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:13.934795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.934780    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:13.935032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.934903    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.935101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.935043    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.935447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.935422    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:13.935586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.935449    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:13.937050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.937022    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:13.937499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.937474    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:13.937615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.937597    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "kube-api-access-fl2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:13.961176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:13.961137    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:14.009220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.009177    6479 generic.go:356] "Generic (PLEG): container finished" podID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerID="1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829" exitCode=0
Nov 01 20:32:14.009418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.009220    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" event={"ID":"c6263de7-73f1-434c-bf75-230cc71de2f8","Type":"ContainerDied","Data":"1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829"}
Nov 01 20:32:14.009418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.009260    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"
Nov 01 20:32:14.009418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.009333    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk" event={"ID":"c6263de7-73f1-434c-bf75-230cc71de2f8","Type":"ContainerDied","Data":"8fe11b5b2f3a4628961e34bcef55d7d515070c533a7f8e6cfe25a33e1881d960"}
Nov 01 20:32:14.009418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.009377    6479 scope.go:122] "RemoveContainer" containerID="1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829"
Nov 01 20:32:14.022256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.022216    6479 scope.go:122] "RemoveContainer" containerID="4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"
Nov 01 20:32:14.032732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.032710    6479 scope.go:122] "RemoveContainer" containerID="1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829"
Nov 01 20:32:14.033057 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:14.033034    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829\": container with ID starting with 1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829 not found: ID does not exist" containerID="1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829"
Nov 01 20:32:14.033129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.033069    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829"} err="failed to get container status \"1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829\": rpc error: code = NotFound desc = could not find container \"1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829\": container with ID starting with 1f77f57f17f456c8e3df629fd03ebe6cbabf7fd270e9eff3864e4a142b9f5829 not found: ID does not exist"
Nov 01 20:32:14.033129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.033090    6479 scope.go:122] "RemoveContainer" containerID="4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"
Nov 01 20:32:14.033377 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:14.033355    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5\": container with ID starting with 4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5 not found: ID does not exist" containerID="4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"
Nov 01 20:32:14.033437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.033389    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5"} err="failed to get container status \"4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5\": rpc error: code = NotFound desc = could not find container \"4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5\": container with ID starting with 4d97db5acf58d71ddb942b6bb36680bf769f2c069226b64358c9200ffb8199c5 not found: ID does not exist"
Nov 01 20:32:14.035871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.035850    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:14.035968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.035891    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:14.035968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.035936    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:14.036077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.035970    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert\") pod \"c6263de7-73f1-434c-bf75-230cc71de2f8\" (UID: \"c6263de7-73f1-434c-bf75-230cc71de2f8\") "
Nov 01 20:32:14.036200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036161    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:14.036247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036195    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:14.036377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036356    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:14.036512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036498    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.036549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036515    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.036549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036524    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6263de7-73f1-434c-bf75-230cc71de2f8-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.036549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036533    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.036549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036541    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.036695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.036550    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl2sg\" (UniqueName: \"kubernetes.io/projected/c6263de7-73f1-434c-bf75-230cc71de2f8-kube-api-access-fl2sg\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.037890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.037870    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy" pod "c6263de7-73f1-434c-bf75-230cc71de2f8" (UID: "c6263de7-73f1-434c-bf75-230cc71de2f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:14.137336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.137250    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6263de7-73f1-434c-bf75-230cc71de2f8-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.331550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.331516    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"]
Nov 01 20:32:14.335006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.334976    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-fhjdk"]
Nov 01 20:32:14.708331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.708288    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" path="/var/lib/kubelet/pods/c6263de7-73f1-434c-bf75-230cc71de2f8/volumes"
Nov 01 20:32:14.733434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:14.733396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:15.429532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:15.429491    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"]
Nov 01 20:32:15.482640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:15.482601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:15.582642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:15.582607    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:15.782790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:15.782751    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:16.004324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:16.004282    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:32:16.232450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:16.232403    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:16.942446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:16.942414    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:17.360965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:17.360927    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:17.574786 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:17.574748    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:17.814559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:17.814509    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:17.814559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:17.814557    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:18.013465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.013426    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"]
Nov 01 20:32:18.013952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.013727    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j" podUID="462fa0ea-b9e2-4515-89b6-bc93de691a95" containerName="maas-api" containerID="cri-o://15a24c6349570e44fe063516a89921df66da80bf3dcf5307016be7d9d382006b" gracePeriod=30
Nov 01 20:32:18.054748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.054714    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:18.302582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.302549    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:32:18.387129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.387096    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca\") pod \"462fa0ea-b9e2-4515-89b6-bc93de691a95\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") "
Nov 01 20:32:18.387262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.387210    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs\") pod \"462fa0ea-b9e2-4515-89b6-bc93de691a95\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") "
Nov 01 20:32:18.387262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.387239    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls\") pod \"462fa0ea-b9e2-4515-89b6-bc93de691a95\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") "
Nov 01 20:32:18.387499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.387478    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv\" (UniqueName: \"kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv\") pod \"462fa0ea-b9e2-4515-89b6-bc93de691a95\" (UID: \"462fa0ea-b9e2-4515-89b6-bc93de691a95\") "
Nov 01 20:32:18.388155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.388096    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca" pod "462fa0ea-b9e2-4515-89b6-bc93de691a95" (UID: "462fa0ea-b9e2-4515-89b6-bc93de691a95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:18.389523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.389032    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462fa0ea-b9e2-4515-89b6-bc93de691a95-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.392257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.392227    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs" pod "462fa0ea-b9e2-4515-89b6-bc93de691a95" (UID: "462fa0ea-b9e2-4515-89b6-bc93de691a95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:18.392690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.392595    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls" pod "462fa0ea-b9e2-4515-89b6-bc93de691a95" (UID: "462fa0ea-b9e2-4515-89b6-bc93de691a95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:18.392932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.392905    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv" pod "462fa0ea-b9e2-4515-89b6-bc93de691a95" (UID: "462fa0ea-b9e2-4515-89b6-bc93de691a95"). InnerVolumeSpecName "kube-api-access-bhdfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:18.398519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.398496    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:32:18.428762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.428711    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:18.489382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489345    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489410    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489476    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489535    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489671    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489779    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489777    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.489978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489824    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489870    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.489978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.489918    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket\") pod \"f4615003-28af-41a2-a79e-416ce58f2be7\" (UID: \"f4615003-28af-41a2-a79e-416ce58f2be7\") "
Nov 01 20:32:18.490192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.490131    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.490361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.490339    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.490487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.490461    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:18.490574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.490554    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.490983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.490959    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.490987    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdfv\" (UniqueName: \"kubernetes.io/projected/462fa0ea-b9e2-4515-89b6-bc93de691a95-kube-api-access-bhdfv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.491004    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4615003-28af-41a2-a79e-416ce58f2be7-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.491018    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.491033    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.491048    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.491063    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462fa0ea-b9e2-4515-89b6-bc93de691a95-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.491080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.491077    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.492192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.492162    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:18.492322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.492274    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:18.492322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.492284    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.492493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.492471    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76" pod "f4615003-28af-41a2-a79e-416ce58f2be7" (UID: "f4615003-28af-41a2-a79e-416ce58f2be7"). InnerVolumeSpecName "kube-api-access-rkz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:18.592537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.592500    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4615003-28af-41a2-a79e-416ce58f2be7-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.592537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.592534    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkz76\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-kube-api-access-rkz76\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.592768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.592546    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4615003-28af-41a2-a79e-416ce58f2be7-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.592768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.592556    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4615003-28af-41a2-a79e-416ce58f2be7-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.688826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.688732    6479 scope.go:122] "RemoveContainer" containerID="a132552396c0291abae65c540214b828b405786aa338494db644f46cfebd18ba"
Nov 01 20:32:18.702419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:18.702394    6479 scope.go:122] "RemoveContainer" containerID="15a24c6349570e44fe063516a89921df66da80bf3dcf5307016be7d9d382006b"
Nov 01 20:32:19.051320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.048960    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j" event={"ID":"462fa0ea-b9e2-4515-89b6-bc93de691a95","Type":"ContainerDied","Data":"15a24c6349570e44fe063516a89921df66da80bf3dcf5307016be7d9d382006b"}
Nov 01 20:32:19.051320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.049023    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"
Nov 01 20:32:19.051320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.049053    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j" event={"ID":"462fa0ea-b9e2-4515-89b6-bc93de691a95","Type":"ContainerDied","Data":"788944ed943d2216e92c2e7ef986f23d93206e7079965a7c80426fa7b8baec9c"}
Nov 01 20:32:19.051320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.049086    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" event={"ID":"f4615003-28af-41a2-a79e-416ce58f2be7","Type":"ContainerDied","Data":"a132552396c0291abae65c540214b828b405786aa338494db644f46cfebd18ba"}
Nov 01 20:32:19.051320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.049119    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x" event={"ID":"f4615003-28af-41a2-a79e-416ce58f2be7","Type":"ContainerDied","Data":"1b5aa4461d778dc2e3b9231e4edfc022598c69f565c25402992d4f0491b30450"}
Nov 01 20:32:19.051320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.049192    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"
Nov 01 20:32:19.069983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.069926    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:19.166742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.166122    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"]
Nov 01 20:32:19.166742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.166622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:19.174269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.174240    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-036c69-77cb664f67-4zd7j"]
Nov 01 20:32:19.178030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.177999    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"]
Nov 01 20:32:19.195202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.195170    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-dg96x"]
Nov 01 20:32:19.827469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.827427    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:19.931741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:19.931708    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:20.027488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.027447    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:20.430339 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.430297    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerName="payload-pre-processing" containerID="cri-o://b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772" gracePeriod=25
Nov 01 20:32:20.445262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.445227    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:20.545323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.545281    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:20.736857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.736805    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462fa0ea-b9e2-4515-89b6-bc93de691a95" path="/var/lib/kubelet/pods/462fa0ea-b9e2-4515-89b6-bc93de691a95/volumes"
Nov 01 20:32:20.737578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.737532    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" path="/var/lib/kubelet/pods/f4615003-28af-41a2-a79e-416ce58f2be7/volumes"
Nov 01 20:32:20.814872 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.814844    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:32:20.848141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.848116    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:32:20.939951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.939914    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj\" (UniqueName: \"kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj\") pod \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") "
Nov 01 20:32:20.940170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.939975    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd\" (UniqueName: \"kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd\") pod \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") "
Nov 01 20:32:20.940170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.940076    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume\") pod \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\" (UID: \"cc2f7d92-05fc-4a3d-95ba-beea9241bab6\") "
Nov 01 20:32:20.940170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.940139    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume\") pod \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\" (UID: \"1cce8947-6a76-450d-b0d9-38d96efe2a0c\") "
Nov 01 20:32:20.940541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.940504    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume" pod "cc2f7d92-05fc-4a3d-95ba-beea9241bab6" (UID: "cc2f7d92-05fc-4a3d-95ba-beea9241bab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:20.940685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.940558    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume" pod "1cce8947-6a76-450d-b0d9-38d96efe2a0c" (UID: "1cce8947-6a76-450d-b0d9-38d96efe2a0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:20.942204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.942182    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd" pod "1cce8947-6a76-450d-b0d9-38d96efe2a0c" (UID: "1cce8947-6a76-450d-b0d9-38d96efe2a0c"). InnerVolumeSpecName "kube-api-access-cg7jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:20.942292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:20.942204    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj" pod "cc2f7d92-05fc-4a3d-95ba-beea9241bab6" (UID: "cc2f7d92-05fc-4a3d-95ba-beea9241bab6"). InnerVolumeSpecName "kube-api-access-92qpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:21.046798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.046693    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.046798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.046733    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cce8947-6a76-450d-b0d9-38d96efe2a0c-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.046798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.046758    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qpj\" (UniqueName: \"kubernetes.io/projected/cc2f7d92-05fc-4a3d-95ba-beea9241bab6-kube-api-access-92qpj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.046798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.046774    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7jd\" (UniqueName: \"kubernetes.io/projected/1cce8947-6a76-450d-b0d9-38d96efe2a0c-kube-api-access-cg7jd\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.047710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.047636    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:21.076715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.076680    6479 generic.go:356] "Generic (PLEG): container finished" podID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerID="5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570" exitCode=0
Nov 01 20:32:21.076895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.076735    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"
Nov 01 20:32:21.076895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.076741    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" event={"ID":"1cce8947-6a76-450d-b0d9-38d96efe2a0c","Type":"ContainerDied","Data":"5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570"}
Nov 01 20:32:21.076895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.076827    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj" event={"ID":"1cce8947-6a76-450d-b0d9-38d96efe2a0c","Type":"ContainerDied","Data":"05a4b78a892c94f9e7203da8a2d3a51e8b6ccd89ce912fdafb3583caeffb2c53"}
Nov 01 20:32:21.076895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.076857    6479 scope.go:122] "RemoveContainer" containerID="5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570"
Nov 01 20:32:21.078575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.078543    6479 generic.go:356] "Generic (PLEG): container finished" podID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerID="0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32" exitCode=0
Nov 01 20:32:21.078698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.078581    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" event={"ID":"cc2f7d92-05fc-4a3d-95ba-beea9241bab6","Type":"ContainerDied","Data":"0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32"}
Nov 01 20:32:21.078756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.078685    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm" event={"ID":"cc2f7d92-05fc-4a3d-95ba-beea9241bab6","Type":"ContainerDied","Data":"fc40b6550528eb068e096874c4ff6b46053cb5e616eb845a4cd6f3b36117f080"}
Nov 01 20:32:21.078901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.078881    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"
Nov 01 20:32:21.090208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.090187    6479 scope.go:122] "RemoveContainer" containerID="5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570"
Nov 01 20:32:21.090546 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:21.090525    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570\": container with ID starting with 5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570 not found: ID does not exist" containerID="5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570"
Nov 01 20:32:21.090640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.090558    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570"} err="failed to get container status \"5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570\": rpc error: code = NotFound desc = could not find container \"5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570\": container with ID starting with 5ccf1409131f62ecda3d7fc98bb0ceecb4cd55828c1c0bcc1b16b1e9aff93570 not found: ID does not exist"
Nov 01 20:32:21.090640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.090576    6479 scope.go:122] "RemoveContainer" containerID="0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32"
Nov 01 20:32:21.100535 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.100516    6479 scope.go:122] "RemoveContainer" containerID="0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32"
Nov 01 20:32:21.100858 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:21.100824    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32\": container with ID starting with 0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32 not found: ID does not exist" containerID="0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32"
Nov 01 20:32:21.100948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.100854    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32"} err="failed to get container status \"0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32\": rpc error: code = NotFound desc = could not find container \"0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32\": container with ID starting with 0999bfb80cad458f7c855e80c47df4bea8c7b6c33f5e7af5bc1ed2c46d445b32 not found: ID does not exist"
Nov 01 20:32:21.109051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.109026    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"]
Nov 01 20:32:21.112424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.112394    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8b6cd6-57c75d8679-zpmjj"]
Nov 01 20:32:21.123317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.123289    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"]
Nov 01 20:32:21.127178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.127130    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:21.127388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.127355    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8b6cd6-d9984647f-7jmjm"]
Nov 01 20:32:21.652689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.652620    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:21.756765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:21.754430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:22.038383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.038346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:22.333505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.333421    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"]
Nov 01 20:32:22.333780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.333750    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy" containerID="cri-o://6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43" gracePeriod=30
Nov 01 20:32:22.357694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.339226    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"]
Nov 01 20:32:22.383311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.383272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:22.468020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.467981    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:22.621748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.620636    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:22.713064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.713029    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" path="/var/lib/kubelet/pods/1cce8947-6a76-450d-b0d9-38d96efe2a0c/volumes"
Nov 01 20:32:22.713544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.713431    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" path="/var/lib/kubelet/pods/cc2f7d92-05fc-4a3d-95ba-beea9241bab6/volumes"
Nov 01 20:32:22.745441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.745398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:22.867575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:22.867540    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:23.039352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:23.039314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:23.139050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:23.138987    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:23.437133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:23.437041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:24.152941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:24.152900    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:24.814206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:24.814174    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:25.478793 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:25.478747    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:26.013671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:26.013606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:26.133556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:26.133517    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:26.745377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:26.745337    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:27.345438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.345397    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:27.598967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.598904    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:32:27.660245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660201    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660300    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660328    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660377    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660488    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660500    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.660606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660558    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660613    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660676    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.660792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660696    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.660792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660726    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token\") pod \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\" (UID: \"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9\") "
Nov 01 20:32:27.661009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.660891    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.661070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.661046    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.661313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.661287    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:27.661391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.661298    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.661391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.661341    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.661391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.661359    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.661391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.661375    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.662970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.662941    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "kube-api-access-67qkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:27.663121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.663094    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:27.663619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.663593    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.664025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.664005    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token" pod "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" (UID: "a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:27.763631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.763309    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.763631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.763345    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qkj\" (UniqueName: \"kubernetes.io/projected/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-kube-api-access-67qkj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.763631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.763368    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.763631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.763382    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.763631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.763397    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.947991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.947961    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:27.994375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:27.994340    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"]
Nov 01 20:32:28.048754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.048708    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:28.125658 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.125614    6479 generic.go:356] "Generic (PLEG): container finished" podID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerID="6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43" exitCode=0
Nov 01 20:32:28.125861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.125686    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"
Nov 01 20:32:28.125861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.125704    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" event={"ID":"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9","Type":"ContainerDied","Data":"6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43"}
Nov 01 20:32:28.125861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.125763    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b" event={"ID":"a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9","Type":"ContainerDied","Data":"3874fe046bd71fa4f42a956a6db4fd92f1003b39e4d1185cb15caa7dcf9040de"}
Nov 01 20:32:28.125861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.125786    6479 scope.go:122] "RemoveContainer" containerID="6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43"
Nov 01 20:32:28.135947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.135930    6479 scope.go:122] "RemoveContainer" containerID="6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43"
Nov 01 20:32:28.136251 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:28.136231    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43\": container with ID starting with 6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43 not found: ID does not exist" containerID="6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43"
Nov 01 20:32:28.136301 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.136266    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43"} err="failed to get container status \"6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43\": rpc error: code = NotFound desc = could not find container \"6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43\": container with ID starting with 6eda00e94591082ab71f247d10e328e212ff2a4832f8fd35c2663cb58ac9fa43 not found: ID does not exist"
Nov 01 20:32:28.147981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.147951    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"]
Nov 01 20:32:28.154537 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.154511    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-w726b"]
Nov 01 20:32:28.707682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.707633    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" path="/var/lib/kubelet/pods/a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9/volumes"
Nov 01 20:32:28.911941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.911901    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"]
Nov 01 20:32:28.941867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:28.941801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:29.041996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:29.041904    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:29.247200 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:29.247154    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:32:29.654349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:29.654308    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:30.282435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:30.282403    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:30.539327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:30.539241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:30.639313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:30.639269    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:31.049807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:31.049774    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:31.655108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:31.655068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:31.854474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:31.854418    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:32.507610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:32.507558    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:32.620181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:32.620139    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:32.995717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:32.995672    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w" podUID="9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" containerName="payload-pre-processing" containerID="cri-o://954f7ee743b6156d9018b21fc81ca91a1fc58f1b094d2235a67f749ebf9d3ab7" gracePeriod=25
Nov 01 20:32:33.038744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.038709    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:32:33.162788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.162568    6479 generic.go:356] "Generic (PLEG): container finished" podID="9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" containerID="954f7ee743b6156d9018b21fc81ca91a1fc58f1b094d2235a67f749ebf9d3ab7" exitCode=0
Nov 01 20:32:33.162788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.162724    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w" event={"ID":"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d","Type":"ContainerDied","Data":"954f7ee743b6156d9018b21fc81ca91a1fc58f1b094d2235a67f749ebf9d3ab7"}
Nov 01 20:32:33.263435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.263403    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:33.273354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.273323    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:32:33.367045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.366143    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume\") pod \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") "
Nov 01 20:32:33.367045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.366286    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p\" (UniqueName: \"kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p\") pod \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\" (UID: \"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d\") "
Nov 01 20:32:33.369190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.369157    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume" pod "9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" (UID: "9f79e45c-d324-43ef-a7cd-efedb4fd9a8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:33.396562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.396506    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p" pod "9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" (UID: "9f79e45c-d324-43ef-a7cd-efedb4fd9a8d"). InnerVolumeSpecName "kube-api-access-rg59p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:33.468393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.468352    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:33.468393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.468395    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg59p\" (UniqueName: \"kubernetes.io/projected/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d-kube-api-access-rg59p\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:33.912761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:33.912690    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8" podUID="78512e90-9f27-4cee-8655-c4bcb41ea1ea" containerName="payload-processing" containerID="cri-o://44bebd3002e5150c2ea4dd8b20880a2a88eaf330c154c08fd42ce796c31385e5" gracePeriod=25
Nov 01 20:32:34.058948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.058903    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:34.182225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.181964    6479 generic.go:356] "Generic (PLEG): container finished" podID="78512e90-9f27-4cee-8655-c4bcb41ea1ea" containerID="44bebd3002e5150c2ea4dd8b20880a2a88eaf330c154c08fd42ce796c31385e5" exitCode=0
Nov 01 20:32:34.182225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.182032    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8" event={"ID":"78512e90-9f27-4cee-8655-c4bcb41ea1ea","Type":"ContainerDied","Data":"44bebd3002e5150c2ea4dd8b20880a2a88eaf330c154c08fd42ce796c31385e5"}
Nov 01 20:32:34.184586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.184545    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w" event={"ID":"9f79e45c-d324-43ef-a7cd-efedb4fd9a8d","Type":"ContainerDied","Data":"3d790533fd8a178ea1243dce327b903d6bf415a8c1bb816c8ad977da95dbf64d"}
Nov 01 20:32:34.185125 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.185018    6479 scope.go:122] "RemoveContainer" containerID="954f7ee743b6156d9018b21fc81ca91a1fc58f1b094d2235a67f749ebf9d3ab7"
Nov 01 20:32:34.185931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.185908    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"
Nov 01 20:32:34.232839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.232817    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:32:34.242728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.242697    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"]
Nov 01 20:32:34.246945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.246913    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a6873b-79d848b447-zbs7w"]
Nov 01 20:32:34.281271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.281237    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk\" (UniqueName: \"kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk\") pod \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") "
Nov 01 20:32:34.281471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.281276    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume\") pod \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\" (UID: \"78512e90-9f27-4cee-8655-c4bcb41ea1ea\") "
Nov 01 20:32:34.281808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.281781    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume" pod "78512e90-9f27-4cee-8655-c4bcb41ea1ea" (UID: "78512e90-9f27-4cee-8655-c4bcb41ea1ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:34.283382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.283347    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk" pod "78512e90-9f27-4cee-8655-c4bcb41ea1ea" (UID: "78512e90-9f27-4cee-8655-c4bcb41ea1ea"). InnerVolumeSpecName "kube-api-access-wdwrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:34.382257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.382216    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwrk\" (UniqueName: \"kubernetes.io/projected/78512e90-9f27-4cee-8655-c4bcb41ea1ea-kube-api-access-wdwrk\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:34.382257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.382253    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78512e90-9f27-4cee-8655-c4bcb41ea1ea-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:34.708921 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.708879    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" path="/var/lib/kubelet/pods/9f79e45c-d324-43ef-a7cd-efedb4fd9a8d/volumes"
Nov 01 20:32:34.736177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:34.736146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:35.192423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:35.192331    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8" event={"ID":"78512e90-9f27-4cee-8655-c4bcb41ea1ea","Type":"ContainerDied","Data":"616c922b0c5283481993f6ac85d8ae4de78ca1b0c70a09ecfd7ae5c6807061f3"}
Nov 01 20:32:35.192423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:35.192360    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"
Nov 01 20:32:35.192423 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:35.192403    6479 scope.go:122] "RemoveContainer" containerID="44bebd3002e5150c2ea4dd8b20880a2a88eaf330c154c08fd42ce796c31385e5"
Nov 01 20:32:35.210397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:35.210375    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"]
Nov 01 20:32:35.214035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:35.214010    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-29b57c-85958bc8ff-7zkd8"]
Nov 01 20:32:35.342084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:35.342051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:36.016188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:36.016149    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:36.639631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:36.639583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:36.723235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:36.723197    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78512e90-9f27-4cee-8655-c4bcb41ea1ea" path="/var/lib/kubelet/pods/78512e90-9f27-4cee-8655-c4bcb41ea1ea/volumes"
Nov 01 20:32:36.740680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:36.740627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:36.849286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:36.847934    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:37.322113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.322075    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:37.371610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.371565    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" podUID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" containerName="main" containerID="cri-o://01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323" gracePeriod=45
Nov 01 20:32:37.457397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.457349    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:37.680891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.680532    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:32:37.723232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.723193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:37.835449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835374    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq\" (UniqueName: \"kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq\") pod \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") "
Nov 01 20:32:37.835449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835420    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache\") pod \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") "
Nov 01 20:32:37.835449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835447    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs\") pod \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") "
Nov 01 20:32:37.835724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835508    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home\") pod \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") "
Nov 01 20:32:37.835724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835570    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm\") pod \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") "
Nov 01 20:32:37.835724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835693    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache" pod "10f7f2ba-8bf9-470e-b738-000dcb16f3c4" (UID: "10f7f2ba-8bf9-470e-b738-000dcb16f3c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.835894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835701    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir\") pod \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\" (UID: \"10f7f2ba-8bf9-470e-b738-000dcb16f3c4\") "
Nov 01 20:32:37.835894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835881    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home" pod "10f7f2ba-8bf9-470e-b738-000dcb16f3c4" (UID: "10f7f2ba-8bf9-470e-b738-000dcb16f3c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.836014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.835904    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir" pod "10f7f2ba-8bf9-470e-b738-000dcb16f3c4" (UID: "10f7f2ba-8bf9-470e-b738-000dcb16f3c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.836172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.836147    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.836172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.836168    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.836172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.836178    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.837766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.837742    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs" pod "10f7f2ba-8bf9-470e-b738-000dcb16f3c4" (UID: "10f7f2ba-8bf9-470e-b738-000dcb16f3c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:37.837868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.837762    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm" pod "10f7f2ba-8bf9-470e-b738-000dcb16f3c4" (UID: "10f7f2ba-8bf9-470e-b738-000dcb16f3c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.837912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.837878    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq" pod "10f7f2ba-8bf9-470e-b738-000dcb16f3c4" (UID: "10f7f2ba-8bf9-470e-b738-000dcb16f3c4"). InnerVolumeSpecName "kube-api-access-rvvqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:37.937899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.937854    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.937899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.937889    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvqq\" (UniqueName: \"kubernetes.io/projected/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-kube-api-access-rvvqq\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.937899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:37.937905    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f7f2ba-8bf9-470e-b738-000dcb16f3c4-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:38.212583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.212544    6479 generic.go:356] "Generic (PLEG): container finished" podID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" containerID="01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323" exitCode=0
Nov 01 20:32:38.212806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.212591    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"
Nov 01 20:32:38.212806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.212594    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" event={"ID":"10f7f2ba-8bf9-470e-b738-000dcb16f3c4","Type":"ContainerDied","Data":"01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323"}
Nov 01 20:32:38.212806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.212707    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96" event={"ID":"10f7f2ba-8bf9-470e-b738-000dcb16f3c4","Type":"ContainerDied","Data":"cf7a7fc1edc64851a1e992f46e5161df6ad2e0ba2f87c5da348f3644d260b127"}
Nov 01 20:32:38.212806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.212736    6479 scope.go:122] "RemoveContainer" containerID="01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323"
Nov 01 20:32:38.223136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.223104    6479 scope.go:122] "RemoveContainer" containerID="01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323"
Nov 01 20:32:38.223464 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:38.223442    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323\": container with ID starting with 01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323 not found: ID does not exist" containerID="01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323"
Nov 01 20:32:38.223536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.223472    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323"} err="failed to get container status \"01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323\": rpc error: code = NotFound desc = could not find container \"01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323\": container with ID starting with 01c178e577a5a4f66eaeabd33ff54e8c29dbdf11b851a70630c5e0b37f754323 not found: ID does not exist"
Nov 01 20:32:38.234254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.234228    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"]
Nov 01 20:32:38.239754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.239728    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/premium-w1-a6873b-kserve-95dff6f9b-sxs96"]
Nov 01 20:32:38.254661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.254625    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:38.472420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.472323    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:38.707020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.706988    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" path="/var/lib/kubelet/pods/10f7f2ba-8bf9-470e-b738-000dcb16f3c4/volumes"
Nov 01 20:32:38.852386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:38.852276    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:39.440576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:39.440546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:40.047615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.047558    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:40.642917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.642865    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:40.712051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.712004    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:32:40.870895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.870857    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume\") pod \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") "
Nov 01 20:32:40.871103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.870941    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p\" (UniqueName: \"kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p\") pod \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\" (UID: \"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53\") "
Nov 01 20:32:40.871274 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.871245    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume" pod "63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" (UID: "63f68aa6-d7bc-4230-b5ea-6bcf505a7f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:40.871470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.871451    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:40.873087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.873059    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p" pod "63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" (UID: "63f68aa6-d7bc-4230-b5ea-6bcf505a7f53"). InnerVolumeSpecName "kube-api-access-6g89p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:40.980637 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:40.973618    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g89p\" (UniqueName: \"kubernetes.io/projected/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53-kube-api-access-6g89p\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:41.232930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.232839    6479 generic.go:356] "Generic (PLEG): container finished" podID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerID="b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772" exitCode=0
Nov 01 20:32:41.232930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.232898    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"
Nov 01 20:32:41.232930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.232917    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" event={"ID":"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53","Type":"ContainerDied","Data":"b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772"}
Nov 01 20:32:41.233555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.232991    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt" event={"ID":"63f68aa6-d7bc-4230-b5ea-6bcf505a7f53","Type":"ContainerDied","Data":"863e9c3d1574710ff0a48d0816792199be52ad7443ea0363c7c128d422c732b1"}
Nov 01 20:32:41.233555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.233014    6479 scope.go:122] "RemoveContainer" containerID="b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772"
Nov 01 20:32:41.243552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.243534    6479 scope.go:122] "RemoveContainer" containerID="b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772"
Nov 01 20:32:41.243858 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:41.243834    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772\": container with ID starting with b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772 not found: ID does not exist" containerID="b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772"
Nov 01 20:32:41.243941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.243866    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772"} err="failed to get container status \"b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772\": rpc error: code = NotFound desc = could not find container \"b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772\": container with ID starting with b76fe5736ba78bfc577afb8d85d8aef59eef161a3e1b5cd372267bd030c7b772 not found: ID does not exist"
Nov 01 20:32:41.254804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.254772    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"]
Nov 01 20:32:41.260899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.260868    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-036c69-7d7b5cd59-5vqnt"]
Nov 01 20:32:41.353052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:41.353016    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:42.038151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:42.038117    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:42.196891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:42.196851    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:42.667895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:42.667854    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:42.706821 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:42.706790    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" path="/var/lib/kubelet/pods/63f68aa6-d7bc-4230-b5ea-6bcf505a7f53/volumes"
Nov 01 20:32:43.262802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:43.262756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:43.853864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:43.853831    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:44.159610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:44.159526    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:32:44.649866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:44.649831    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:45.093710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:45.093676    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:45.152682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:45.149794    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"]
Nov 01 20:32:45.223668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:45.223616    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:45.307875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:45.307831    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:45.539862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:45.539823    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:46.246170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:46.246127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:46.847298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:46.847258    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:47.152683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:47.152584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:47.530952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:47.530919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:47.982098 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:47.982058    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"]
Nov 01 20:32:47.982392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:47.982355    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" podUID="cf650dd7-0480-4237-9ac5-2267acad84d5" containerName="maas-api" containerID="cri-o://0f9ec6c3f65dcde12b5d81bfaab206b14e8b937e596a9db76a252a8aba3186fd" gracePeriod=30
Nov 01 20:32:48.022236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.022194    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:48.286292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.285993    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:48.289765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.289630    6479 generic.go:356] "Generic (PLEG): container finished" podID="cf650dd7-0480-4237-9ac5-2267acad84d5" containerID="0f9ec6c3f65dcde12b5d81bfaab206b14e8b937e596a9db76a252a8aba3186fd" exitCode=0
Nov 01 20:32:48.289930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.289702    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" event={"ID":"cf650dd7-0480-4237-9ac5-2267acad84d5","Type":"ContainerDied","Data":"0f9ec6c3f65dcde12b5d81bfaab206b14e8b937e596a9db76a252a8aba3186fd"}
Nov 01 20:32:48.336160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.336133    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:32:48.484736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.484697    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca\") pod \"cf650dd7-0480-4237-9ac5-2267acad84d5\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") "
Nov 01 20:32:48.484956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.484839    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4\" (UniqueName: \"kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4\") pod \"cf650dd7-0480-4237-9ac5-2267acad84d5\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") "
Nov 01 20:32:48.484956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.484908    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\") pod \"cf650dd7-0480-4237-9ac5-2267acad84d5\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") "
Nov 01 20:32:48.485097 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.485026    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs\") pod \"cf650dd7-0480-4237-9ac5-2267acad84d5\" (UID: \"cf650dd7-0480-4237-9ac5-2267acad84d5\") "
Nov 01 20:32:48.485187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.485161    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca" pod "cf650dd7-0480-4237-9ac5-2267acad84d5" (UID: "cf650dd7-0480-4237-9ac5-2267acad84d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:48.485660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.485617    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf650dd7-0480-4237-9ac5-2267acad84d5-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:48.487268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.487238    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs" pod "cf650dd7-0480-4237-9ac5-2267acad84d5" (UID: "cf650dd7-0480-4237-9ac5-2267acad84d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:48.487440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.487310    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls" pod "cf650dd7-0480-4237-9ac5-2267acad84d5" (UID: "cf650dd7-0480-4237-9ac5-2267acad84d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:48.487440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.487321    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4" pod "cf650dd7-0480-4237-9ac5-2267acad84d5" (UID: "cf650dd7-0480-4237-9ac5-2267acad84d5"). InnerVolumeSpecName "kube-api-access-f8rr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:48.587420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.587271    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8rr4\" (UniqueName: \"kubernetes.io/projected/cf650dd7-0480-4237-9ac5-2267acad84d5-kube-api-access-f8rr4\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:48.587420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.587314    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:48.587420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:48.587328    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf650dd7-0480-4237-9ac5-2267acad84d5-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:49.053819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.053778    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:49.298044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.298011    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"
Nov 01 20:32:49.298251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.298057    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb" event={"ID":"cf650dd7-0480-4237-9ac5-2267acad84d5","Type":"ContainerDied","Data":"7b211625b11094dac305d54a28e8184d0eedec7c8fdbf7be0c95300e05bf02e5"}
Nov 01 20:32:49.298251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.298126    6479 scope.go:122] "RemoveContainer" containerID="0f9ec6c3f65dcde12b5d81bfaab206b14e8b937e596a9db76a252a8aba3186fd"
Nov 01 20:32:49.314484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.314450    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"]
Nov 01 20:32:49.317884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.317854    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a6873b-85484cd666-29vwb"]
Nov 01 20:32:49.522410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.522366    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"]
Nov 01 20:32:49.522828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.522631    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d" podUID="e39fa649-e41d-4ed0-9858-5024d1fc3a5b" containerName="maas-api" containerID="cri-o://1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e" gracePeriod=30
Nov 01 20:32:49.551332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.551293    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:49.771816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.766598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:49.894348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:49.894278    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:32:50.010804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.010741    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs\") pod \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") "
Nov 01 20:32:50.011032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.010862    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s\" (UniqueName: \"kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s\") pod \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") "
Nov 01 20:32:50.011032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.010904    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls\") pod \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") "
Nov 01 20:32:50.011032 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.010978    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca\") pod \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\" (UID: \"e39fa649-e41d-4ed0-9858-5024d1fc3a5b\") "
Nov 01 20:32:50.011601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.011549    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca" pod "e39fa649-e41d-4ed0-9858-5024d1fc3a5b" (UID: "e39fa649-e41d-4ed0-9858-5024d1fc3a5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:50.013253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.013224    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls" pod "e39fa649-e41d-4ed0-9858-5024d1fc3a5b" (UID: "e39fa649-e41d-4ed0-9858-5024d1fc3a5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:50.013403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.013383    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs" pod "e39fa649-e41d-4ed0-9858-5024d1fc3a5b" (UID: "e39fa649-e41d-4ed0-9858-5024d1fc3a5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:50.013507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.013462    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s" pod "e39fa649-e41d-4ed0-9858-5024d1fc3a5b" (UID: "e39fa649-e41d-4ed0-9858-5024d1fc3a5b"). InnerVolumeSpecName "kube-api-access-4qk7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:50.112720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.112676    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:50.112720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.112711    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:50.112720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.112723    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:50.112978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.112732    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qk7s\" (UniqueName: \"kubernetes.io/projected/e39fa649-e41d-4ed0-9858-5024d1fc3a5b-kube-api-access-4qk7s\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:50.150848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.150739    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" podUID="d3644b62-073a-4e34-acc2-b0657b254966" containerName="payload-pre-processing" containerID="cri-o://57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861" gracePeriod=25
Nov 01 20:32:50.306639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.306595    6479 generic.go:356] "Generic (PLEG): container finished" podID="e39fa649-e41d-4ed0-9858-5024d1fc3a5b" containerID="1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e" exitCode=0
Nov 01 20:32:50.306878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.306680    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"
Nov 01 20:32:50.306878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.306682    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d" event={"ID":"e39fa649-e41d-4ed0-9858-5024d1fc3a5b","Type":"ContainerDied","Data":"1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e"}
Nov 01 20:32:50.306878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.306740    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d" event={"ID":"e39fa649-e41d-4ed0-9858-5024d1fc3a5b","Type":"ContainerDied","Data":"6bd12d94daa387de62f8080495544be7d50af3d24897015beb8b6ed54458b6d8"}
Nov 01 20:32:50.306878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.306763    6479 scope.go:122] "RemoveContainer" containerID="1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e"
Nov 01 20:32:50.317901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.317875    6479 scope.go:122] "RemoveContainer" containerID="1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e"
Nov 01 20:32:50.318242 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:50.318222    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e\": container with ID starting with 1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e not found: ID does not exist" containerID="1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e"
Nov 01 20:32:50.318322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.318250    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e"} err="failed to get container status \"1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e\": rpc error: code = NotFound desc = could not find container \"1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e\": container with ID starting with 1a685e9d4677e4d22749843c0ebf6ec6872a993c94b2558974528e239a48dd1e not found: ID does not exist"
Nov 01 20:32:50.328933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.328898    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"]
Nov 01 20:32:50.332208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.332178    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bccce1-58fd5db6f5-dr95d"]
Nov 01 20:32:50.449289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.449259    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:50.708574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.708486    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf650dd7-0480-4237-9ac5-2267acad84d5" path="/var/lib/kubelet/pods/cf650dd7-0480-4237-9ac5-2267acad84d5/volumes"
Nov 01 20:32:50.709035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.709016    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39fa649-e41d-4ed0-9858-5024d1fc3a5b" path="/var/lib/kubelet/pods/e39fa649-e41d-4ed0-9858-5024d1fc3a5b/volumes"
Nov 01 20:32:50.844065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.844027    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:50.935619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:50.935573    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:51.072485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:51.072396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:51.325836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:51.325745    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:51.617402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:51.617312    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"]
Nov 01 20:32:51.617613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:51.617580    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy" containerID="cri-o://deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee" gracePeriod=30
Nov 01 20:32:51.647338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:51.647301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:51.944360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:51.944275    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:52.101133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:52.101082    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:52.699492 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:52.699457    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:53.459463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:53.459430    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:53.635413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:53.635376    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:32:54.270376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:54.270338    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:54.944424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:54.944384    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:55.647925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:55.647889    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:56.257665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.257619    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:56.536553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.536456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:56.635837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.635802    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:56.859387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.859345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:56.869775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.869749    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:32:56.997873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.997834    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.997969    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998045    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998135    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998183    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998216    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998243    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998273    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998309    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy\") pod \"ff981dbc-864a-40a9-8908-df562cae4c22\" (UID: \"ff981dbc-864a-40a9-8908-df562cae4c22\") "
Nov 01 20:32:56.998566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998487    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:56.998641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998610    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:56.998743 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998499    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:56.998913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998854    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:56.998913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998862    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:56.998913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998882    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:56.998913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998898    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:56.999302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:56.998982    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:57.000591 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.000560    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:57.000750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.000596    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:57.000851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.000825    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "kube-api-access-9pqdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:57.000851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.000840    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy" pod "ff981dbc-864a-40a9-8908-df562cae4c22" (UID: "ff981dbc-864a-40a9-8908-df562cae4c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:57.064479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.064441    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"]
Nov 01 20:32:57.079463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.079432    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"]
Nov 01 20:32:57.099862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.099767    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pqdr\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-kube-api-access-9pqdr\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:57.099862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.099798    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff981dbc-864a-40a9-8908-df562cae4c22-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:57.099862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.099809    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:57.099862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.099818    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff981dbc-864a-40a9-8908-df562cae4c22-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:57.099862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.099827    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff981dbc-864a-40a9-8908-df562cae4c22-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:57.099862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.099839    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff981dbc-864a-40a9-8908-df562cae4c22-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:57.159877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.159843    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:57.369165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.369079    6479 generic.go:356] "Generic (PLEG): container finished" podID="ff981dbc-864a-40a9-8908-df562cae4c22" containerID="deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee" exitCode=0
Nov 01 20:32:57.369626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.369117    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" event={"ID":"ff981dbc-864a-40a9-8908-df562cae4c22","Type":"ContainerDied","Data":"deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee"}
Nov 01 20:32:57.369811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.369699    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8" event={"ID":"ff981dbc-864a-40a9-8908-df562cae4c22","Type":"ContainerDied","Data":"5b60a098bd6d96757ce0685bb6d6b892eee043fae761ab19c31924442c38c5eb"}
Nov 01 20:32:57.369811 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.369741    6479 scope.go:122] "RemoveContainer" containerID="deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee"
Nov 01 20:32:57.369939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.369159    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"
Nov 01 20:32:57.380972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.380943    6479 scope.go:122] "RemoveContainer" containerID="deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee"
Nov 01 20:32:57.381313 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:57.381292    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee\": container with ID starting with deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee not found: ID does not exist" containerID="deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee"
Nov 01 20:32:57.381376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.381324    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee"} err="failed to get container status \"deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee\": rpc error: code = NotFound desc = could not find container \"deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee\": container with ID starting with deabd9fa3a303e5d338912c9f0edf7d6c139481a730b985abd1af5af088791ee not found: ID does not exist"
Nov 01 20:32:57.393899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.393718    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"]
Nov 01 20:32:57.396629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.396593    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-gznb8"]
Nov 01 20:32:57.566409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.566367    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:57.664505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:57.664416    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:58.005091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:58.005045    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" podUID="d3644b62-073a-4e34-acc2-b0657b254966" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:58.310104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:58.310011    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:58.708297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:58.708263    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" path="/var/lib/kubelet/pods/ff981dbc-864a-40a9-8908-df562cae4c22/volumes"
Nov 01 20:32:58.969459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:58.969364    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:59.175563 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:32:59.175515    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:32:59.552306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:59.552271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:59.652608 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:59.652567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:32:59.752354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:32:59.752307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:00.436290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:00.436248    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:33:00.532472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:00.532432    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:00.948015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:00.947927    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:01.308490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:01.308394    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:01.415396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:01.415356    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:01.728308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:01.727747    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"]
Nov 01 20:33:01.728308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:01.728002    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy" containerID="cri-o://e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903" gracePeriod=30
Nov 01 20:33:01.740781 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:01.740749    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:02.232995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:02.232008    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:02.495794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:02.495709    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:02.753109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:02.753025    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:03.359579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:03.359543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:03.459569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:03.459532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:03.559994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:03.559956    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:04.056176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:04.056137    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:04.280123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:04.280085    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:04.969372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:04.969336    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:05.567911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:05.567875    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:06.164662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:06.164609    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:06.672259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:06.672209    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:06.750293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:06.749968    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:06.873615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:06.873584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:07.015749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.015721    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:33:07.123728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.123690    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.123943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.123735    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.123943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.123782    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.123958    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124027    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124058    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.124201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124089    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124165    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124216    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:07.124360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124229    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124342    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.124520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124403    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs\") pod \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\" (UID: \"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c\") "
Nov 01 20:33:07.124520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124440    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.124690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124667    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.124928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124908    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.124984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124934    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.124984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124950    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.124984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124964    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.124984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.124979    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.126370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.126344    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:07.126642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.126620    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "kube-api-access-9kfnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:07.126750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.126670    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:07.126750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.126671    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy" pod "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" (UID: "6c65fc48-3041-4a60-a9da-b7ed8ab4d83c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.226255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.226214    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.226255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.226249    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.226255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.226261    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.226746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.226275    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfnz\" (UniqueName: \"kubernetes.io/projected/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c-kube-api-access-9kfnz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.457425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.457391    6479 generic.go:356] "Generic (PLEG): container finished" podID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerID="e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903" exitCode=0
Nov 01 20:33:07.457670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.457443    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"
Nov 01 20:33:07.457670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.457470    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" event={"ID":"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c","Type":"ContainerDied","Data":"e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903"}
Nov 01 20:33:07.457670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.457533    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2" event={"ID":"6c65fc48-3041-4a60-a9da-b7ed8ab4d83c","Type":"ContainerDied","Data":"af70a6f4751606fec548c53dad42473d82e43c23f1febd97656cb9fd9e86983c"}
Nov 01 20:33:07.457670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.457578    6479 scope.go:122] "RemoveContainer" containerID="e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903"
Nov 01 20:33:07.468884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.468860    6479 scope.go:122] "RemoveContainer" containerID="e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"
Nov 01 20:33:07.479037 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.479013    6479 scope.go:122] "RemoveContainer" containerID="e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903"
Nov 01 20:33:07.479360 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:07.479329    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903\": container with ID starting with e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903 not found: ID does not exist" containerID="e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903"
Nov 01 20:33:07.479520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.479364    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903"} err="failed to get container status \"e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903\": rpc error: code = NotFound desc = could not find container \"e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903\": container with ID starting with e946751b7d626870e0aa881efbdea97e719ea8db77e38a02fd72f14ac8768903 not found: ID does not exist"
Nov 01 20:33:07.479520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.479391    6479 scope.go:122] "RemoveContainer" containerID="e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"
Nov 01 20:33:07.479711 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:07.479689    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06\": container with ID starting with e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06 not found: ID does not exist" containerID="e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"
Nov 01 20:33:07.479772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.479717    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06"} err="failed to get container status \"e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06\": rpc error: code = NotFound desc = could not find container \"e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06\": container with ID starting with e33afa1adad2314ceacb11f5e567cc541ffe587c329d7526b3b511c65196db06 not found: ID does not exist"
Nov 01 20:33:07.479968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.479950    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"]
Nov 01 20:33:07.484912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.484884    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-4d9r2"]
Nov 01 20:33:07.564831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:07.564794    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:08.003932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.003893    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" podUID="d3644b62-073a-4e34-acc2-b0657b254966" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:08.168014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.167947    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:08.658502 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.658464    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"]
Nov 01 20:33:08.678081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.678046    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"]
Nov 01 20:33:08.711091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.711051    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" path="/var/lib/kubelet/pods/6c65fc48-3041-4a60-a9da-b7ed8ab4d83c/volumes"
Nov 01 20:33:08.720440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.720408    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:08.783009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.782964    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"]
Nov 01 20:33:08.783244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.783213    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq" podUID="27c6e266-02c6-4d4c-8b78-e4ba53a3751d" containerName="maas-api" containerID="cri-o://301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea" gracePeriod=30
Nov 01 20:33:08.811521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.811447    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:08.887030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:08.886995    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:09.012684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.011665    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:09.098746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.098715    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:33:09.150963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.150918    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca\") pod \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") "
Nov 01 20:33:09.151127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.151051    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls\") pod \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") "
Nov 01 20:33:09.151127 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.151093    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs\") pod \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") "
Nov 01 20:33:09.151223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.151147    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7\" (UniqueName: \"kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7\") pod \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\" (UID: \"27c6e266-02c6-4d4c-8b78-e4ba53a3751d\") "
Nov 01 20:33:09.151399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.151357    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca" pod "27c6e266-02c6-4d4c-8b78-e4ba53a3751d" (UID: "27c6e266-02c6-4d4c-8b78-e4ba53a3751d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:09.151621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.151601    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:09.153450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.153410    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7" pod "27c6e266-02c6-4d4c-8b78-e4ba53a3751d" (UID: "27c6e266-02c6-4d4c-8b78-e4ba53a3751d"). InnerVolumeSpecName "kube-api-access-4zjl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:09.153450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.153427    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls" pod "27c6e266-02c6-4d4c-8b78-e4ba53a3751d" (UID: "27c6e266-02c6-4d4c-8b78-e4ba53a3751d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:33:09.153692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.153672    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs" pod "27c6e266-02c6-4d4c-8b78-e4ba53a3751d" (UID: "27c6e266-02c6-4d4c-8b78-e4ba53a3751d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:33:09.221969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.221876    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:09.252363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.252324    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:09.252363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.252365    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:09.252577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.252380    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjl7\" (UniqueName: \"kubernetes.io/projected/27c6e266-02c6-4d4c-8b78-e4ba53a3751d-kube-api-access-4zjl7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:09.472315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.472226    6479 generic.go:356] "Generic (PLEG): container finished" podID="27c6e266-02c6-4d4c-8b78-e4ba53a3751d" containerID="301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea" exitCode=0
Nov 01 20:33:09.472315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.472275    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"
Nov 01 20:33:09.472542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.472313    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq" event={"ID":"27c6e266-02c6-4d4c-8b78-e4ba53a3751d","Type":"ContainerDied","Data":"301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea"}
Nov 01 20:33:09.472542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.472372    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq" event={"ID":"27c6e266-02c6-4d4c-8b78-e4ba53a3751d","Type":"ContainerDied","Data":"355eeff9448f8f555802725556cd108c553434f8ef5cb5d8d8f8615fbf1ef66d"}
Nov 01 20:33:09.472542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.472403    6479 scope.go:122] "RemoveContainer" containerID="301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea"
Nov 01 20:33:09.483011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.482992    6479 scope.go:122] "RemoveContainer" containerID="301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea"
Nov 01 20:33:09.483336 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:09.483316    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea\": container with ID starting with 301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea not found: ID does not exist" containerID="301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea"
Nov 01 20:33:09.483421 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.483347    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea"} err="failed to get container status \"301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea\": rpc error: code = NotFound desc = could not find container \"301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea\": container with ID starting with 301ce980f43dc893100118789398f1aafdfeab386eb7f1c92c8103078d8b10ea not found: ID does not exist"
Nov 01 20:33:09.497624 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.497062    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"]
Nov 01 20:33:09.499100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.499067    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-9be196-bfc46bbb-5pzsq"]
Nov 01 20:33:09.858712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:09.858578    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:10.441620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.441594    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:33:10.488690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.488638    6479 generic.go:356] "Generic (PLEG): container finished" podID="d3644b62-073a-4e34-acc2-b0657b254966" containerID="57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861" exitCode=0
Nov 01 20:33:10.488885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.488704    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" event={"ID":"d3644b62-073a-4e34-acc2-b0657b254966","Type":"ContainerDied","Data":"57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861"}
Nov 01 20:33:10.488885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.488717    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"
Nov 01 20:33:10.488885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.488788    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl" event={"ID":"d3644b62-073a-4e34-acc2-b0657b254966","Type":"ContainerDied","Data":"4c1485449eead96e6121a89aba98e79f10823e83f8a9768418321dd22310f696"}
Nov 01 20:33:10.488885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.488820    6479 scope.go:122] "RemoveContainer" containerID="57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861"
Nov 01 20:33:10.500289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.500266    6479 scope.go:122] "RemoveContainer" containerID="57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861"
Nov 01 20:33:10.500655 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:10.500620    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861\": container with ID starting with 57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861 not found: ID does not exist" containerID="57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861"
Nov 01 20:33:10.500790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.500665    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861"} err="failed to get container status \"57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861\": rpc error: code = NotFound desc = could not find container \"57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861\": container with ID starting with 57639cd8d1864d9519c3e9dad224fc081237d06ea87bb60fbb97addc1670c861 not found: ID does not exist"
Nov 01 20:33:10.572389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.572351    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj\" (UniqueName: \"kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj\") pod \"d3644b62-073a-4e34-acc2-b0657b254966\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") "
Nov 01 20:33:10.572389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.572396    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume\") pod \"d3644b62-073a-4e34-acc2-b0657b254966\" (UID: \"d3644b62-073a-4e34-acc2-b0657b254966\") "
Nov 01 20:33:10.572878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.572844    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume" pod "d3644b62-073a-4e34-acc2-b0657b254966" (UID: "d3644b62-073a-4e34-acc2-b0657b254966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:10.574589 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.574565    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj" pod "d3644b62-073a-4e34-acc2-b0657b254966" (UID: "d3644b62-073a-4e34-acc2-b0657b254966"). InnerVolumeSpecName "kube-api-access-khvtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:10.674350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.674251    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khvtj\" (UniqueName: \"kubernetes.io/projected/d3644b62-073a-4e34-acc2-b0657b254966-kube-api-access-khvtj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:10.674350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.674289    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3644b62-073a-4e34-acc2-b0657b254966-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:10.706359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.706321    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c6e266-02c6-4d4c-8b78-e4ba53a3751d" path="/var/lib/kubelet/pods/27c6e266-02c6-4d4c-8b78-e4ba53a3751d/volumes"
Nov 01 20:33:10.808691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.808636    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"]
Nov 01 20:33:10.816623 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.816593    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bccce1-f5ff9c447-whvwl"]
Nov 01 20:33:10.849088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:10.849035    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:11.615574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:11.615515    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:11.716017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:11.715970    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:12.087765 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.087718    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" containerID="cri-o://322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908" gracePeriod=45
Nov 01 20:33:12.106411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.106375    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" containerID="cri-o://094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959" gracePeriod=45
Nov 01 20:33:12.171660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.171617    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": dial tcp 10.134.0.37:8000: connect: connection refused"
Nov 01 20:33:12.218718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.218676    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused"
Nov 01 20:33:12.364859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.364819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:12.381791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.381757    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:33:12.409106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.409076    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:33:12.492780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.492731    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home\") pod \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") "
Nov 01 20:33:12.492986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.492786    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache\") pod \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") "
Nov 01 20:33:12.492986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.492832    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache\") pod \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") "
Nov 01 20:33:12.492986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.492862    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd\" (UniqueName: \"kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd\") pod \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") "
Nov 01 20:33:12.492986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.492930    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir\") pod \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") "
Nov 01 20:33:12.492986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.492977    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs\") pod \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") "
Nov 01 20:33:12.493262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493026    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs\" (UniqueName: \"kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs\") pod \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") "
Nov 01 20:33:12.493262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493021    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home" pod "13f5dfca-b212-43a2-b3ba-4e316cdbbd71" (UID: "13f5dfca-b212-43a2-b3ba-4e316cdbbd71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.493262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493042    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache" pod "07174d89-b04b-47e6-8fd8-e8cfa611c75a" (UID: "07174d89-b04b-47e6-8fd8-e8cfa611c75a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.493262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493055    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm\") pod \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") "
Nov 01 20:33:12.493262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493252    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir" pod "13f5dfca-b212-43a2-b3ba-4e316cdbbd71" (UID: "13f5dfca-b212-43a2-b3ba-4e316cdbbd71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.493534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493310    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache" pod "13f5dfca-b212-43a2-b3ba-4e316cdbbd71" (UID: "13f5dfca-b212-43a2-b3ba-4e316cdbbd71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.493534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493126    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf\" (UniqueName: \"kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf\") pod \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") "
Nov 01 20:33:12.493534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493521    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home\") pod \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") "
Nov 01 20:33:12.493716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493559    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm\") pod \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\" (UID: \"13f5dfca-b212-43a2-b3ba-4e316cdbbd71\") "
Nov 01 20:33:12.493716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493607    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir\") pod \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\" (UID: \"07174d89-b04b-47e6-8fd8-e8cfa611c75a\") "
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.493844    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home" pod "07174d89-b04b-47e6-8fd8-e8cfa611c75a" (UID: "07174d89-b04b-47e6-8fd8-e8cfa611c75a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494046    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir" pod "07174d89-b04b-47e6-8fd8-e8cfa611c75a" (UID: "07174d89-b04b-47e6-8fd8-e8cfa611c75a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494386    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494453    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494468    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494481    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494494    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.494842 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.494511    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.495924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.495898    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs" pod "13f5dfca-b212-43a2-b3ba-4e316cdbbd71" (UID: "13f5dfca-b212-43a2-b3ba-4e316cdbbd71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:33:12.496047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.495939    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs" pod "07174d89-b04b-47e6-8fd8-e8cfa611c75a" (UID: "07174d89-b04b-47e6-8fd8-e8cfa611c75a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:33:12.496265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.496235    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf" pod "07174d89-b04b-47e6-8fd8-e8cfa611c75a" (UID: "07174d89-b04b-47e6-8fd8-e8cfa611c75a"). InnerVolumeSpecName "kube-api-access-8ldlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:12.496369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.496267    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm" pod "07174d89-b04b-47e6-8fd8-e8cfa611c75a" (UID: "07174d89-b04b-47e6-8fd8-e8cfa611c75a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.496822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.496800    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd" pod "13f5dfca-b212-43a2-b3ba-4e316cdbbd71" (UID: "13f5dfca-b212-43a2-b3ba-4e316cdbbd71"). InnerVolumeSpecName "kube-api-access-n5rqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:12.497468 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.497436    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm" pod "13f5dfca-b212-43a2-b3ba-4e316cdbbd71" (UID: "13f5dfca-b212-43a2-b3ba-4e316cdbbd71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:12.502819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.502797    6479 generic.go:356] "Generic (PLEG): container finished" podID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerID="094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959" exitCode=0
Nov 01 20:33:12.502915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.502847    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"
Nov 01 20:33:12.502915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.502882    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" event={"ID":"07174d89-b04b-47e6-8fd8-e8cfa611c75a","Type":"ContainerDied","Data":"094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959"}
Nov 01 20:33:12.503008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.502931    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r" event={"ID":"07174d89-b04b-47e6-8fd8-e8cfa611c75a","Type":"ContainerDied","Data":"a17cb88843c6a9038e26efdaef74b98a3f0b2b176b873d86740ca081b535fae2"}
Nov 01 20:33:12.503008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.502953    6479 scope.go:122] "RemoveContainer" containerID="094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959"
Nov 01 20:33:12.504287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.504264    6479 generic.go:356] "Generic (PLEG): container finished" podID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerID="322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908" exitCode=0
Nov 01 20:33:12.504379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.504295    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" event={"ID":"13f5dfca-b212-43a2-b3ba-4e316cdbbd71","Type":"ContainerDied","Data":"322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908"}
Nov 01 20:33:12.504379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.504322    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"
Nov 01 20:33:12.504379 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.504339    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz" event={"ID":"13f5dfca-b212-43a2-b3ba-4e316cdbbd71","Type":"ContainerDied","Data":"d9d151d7b90e93cf748a1f94e910fc88bb5b830d48074a7ae2560f897846c44b"}
Nov 01 20:33:12.514462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.514439    6479 scope.go:122] "RemoveContainer" containerID="094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959"
Nov 01 20:33:12.514788 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:12.514767    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959\": container with ID starting with 094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959 not found: ID does not exist" containerID="094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959"
Nov 01 20:33:12.514865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.514798    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959"} err="failed to get container status \"094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959\": rpc error: code = NotFound desc = could not find container \"094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959\": container with ID starting with 094a19268a0b5b05717f64590ae60acd6673f537378b9f26c66f488f312dc959 not found: ID does not exist"
Nov 01 20:33:12.514865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.514817    6479 scope.go:122] "RemoveContainer" containerID="322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908"
Nov 01 20:33:12.524837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.524747    6479 scope.go:122] "RemoveContainer" containerID="322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908"
Nov 01 20:33:12.525113 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:12.525088    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908\": container with ID starting with 322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908 not found: ID does not exist" containerID="322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908"
Nov 01 20:33:12.525190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.525120    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908"} err="failed to get container status \"322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908\": rpc error: code = NotFound desc = could not find container \"322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908\": container with ID starting with 322024adc594303d4b8235fa9d90a4700ce3fdc81e09f40922cac5251098f908 not found: ID does not exist"
Nov 01 20:33:12.525611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.525592    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"]
Nov 01 20:33:12.532089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.532064    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/premium-w4-0aa61b-kserve-6b479d4c99-5cs2r"]
Nov 01 20:33:12.540798 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.540773    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"]
Nov 01 20:33:12.544167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.544142    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-2-w4-0aa61b-kserve-6f59b5c577-ffjtz"]
Nov 01 20:33:12.596137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.596092    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.596137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.596137    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07174d89-b04b-47e6-8fd8-e8cfa611c75a-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.596353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.596151    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07174d89-b04b-47e6-8fd8-e8cfa611c75a-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.596353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.596166    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldlf\" (UniqueName: \"kubernetes.io/projected/07174d89-b04b-47e6-8fd8-e8cfa611c75a-kube-api-access-8ldlf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.596353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.596181    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.596353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.596195    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rqd\" (UniqueName: \"kubernetes.io/projected/13f5dfca-b212-43a2-b3ba-4e316cdbbd71-kube-api-access-n5rqd\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:12.706824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.706786    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" path="/var/lib/kubelet/pods/07174d89-b04b-47e6-8fd8-e8cfa611c75a/volumes"
Nov 01 20:33:12.707363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.707343    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" path="/var/lib/kubelet/pods/13f5dfca-b212-43a2-b3ba-4e316cdbbd71/volumes"
Nov 01 20:33:12.707747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.707731    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3644b62-073a-4e34-acc2-b0657b254966" path="/var/lib/kubelet/pods/d3644b62-073a-4e34-acc2-b0657b254966/volumes"
Nov 01 20:33:12.948412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:12.948373    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:13.551641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:13.551605    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:13.659841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:13.659786    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerName="payload-processing" containerID="cri-o://3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c" gracePeriod=25
Nov 01 20:33:13.679419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:13.679365    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" podUID="eede4389-f540-4c70-832e-a55e6198a442" containerName="payload-pre-processing" containerID="cri-o://f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771" gracePeriod=25
Nov 01 20:33:14.154191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:14.154146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:14.319233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:14.319196    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:33:14.870787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:14.870724    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:14.970387 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:14.970352    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:15.364195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:15.364152    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:15.667539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:15.667234    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"]
Nov 01 20:33:15.668180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:15.668139    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy" containerID="cri-o://2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020" gracePeriod=30
Nov 01 20:33:15.697508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:15.697472    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:15.886826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:15.886789    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:16.455424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:16.455380    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:16.680120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:16.680078    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:17.251962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:17.251924    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:17.752463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:17.752421    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:17.912233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:17.912195    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:18.010374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:18.010287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:18.549238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:18.549192    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:19.147703 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:19.147579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:19.599673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:19.595347    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:19.599673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:19.595542    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" podUID="eede4389-f540-4c70-832e-a55e6198a442" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:19.825536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:19.825498    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:20.561831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:20.561794    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:20.927232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:20.927201    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:33:21.114255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114157    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114223    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114257    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114288    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114336    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114449    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114481    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:21.114850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114526    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114580    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114614    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs\") pod \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\" (UID: \"6b718eed-0b7f-48f0-9a27-a03bbd79850c\") "
Nov 01 20:33:21.114850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114613    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:21.115076 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114893    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:21.115076 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.114983    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:21.115200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.115106    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:21.115264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.115204    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.115264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.115230    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.115264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.115246    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.115264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.115260    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.115439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.115275    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.116782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.116753    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "kube-api-access-4ljhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:21.117005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.116985    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:21.117117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.117095    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:21.117204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.117188    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo" pod "6b718eed-0b7f-48f0-9a27-a03bbd79850c" (UID: "6b718eed-0b7f-48f0-9a27-a03bbd79850c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:21.151291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.151250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:21.225975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.225928    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljhz\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-kube-api-access-4ljhz\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.225975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.225976    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.226244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.225991    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.226244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.226007    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b718eed-0b7f-48f0-9a27-a03bbd79850c-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:21.558280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.558244    6479 generic.go:356] "Generic (PLEG): container finished" podID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerID="2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020" exitCode=0
Nov 01 20:33:21.558501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.558299    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"
Nov 01 20:33:21.558501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.558319    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" event={"ID":"6b718eed-0b7f-48f0-9a27-a03bbd79850c","Type":"ContainerDied","Data":"2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020"}
Nov 01 20:33:21.558501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.558366    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh" event={"ID":"6b718eed-0b7f-48f0-9a27-a03bbd79850c","Type":"ContainerDied","Data":"2b48c7328e96368decdf098c2da1325682b9f28929fb957e0ff0706d4815e427"}
Nov 01 20:33:21.558501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.558397    6479 scope.go:122] "RemoveContainer" containerID="2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020"
Nov 01 20:33:21.570206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.570153    6479 scope.go:122] "RemoveContainer" containerID="2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020"
Nov 01 20:33:21.571132 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:21.571108    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020\": container with ID starting with 2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020 not found: ID does not exist" containerID="2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020"
Nov 01 20:33:21.571199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.571142    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020"} err="failed to get container status \"2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020\": rpc error: code = NotFound desc = could not find container \"2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020\": container with ID starting with 2e63259c311962166de050187d3b3b1ee9fe8d050d845f4640f4ebf5d4472020 not found: ID does not exist"
Nov 01 20:33:21.580634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.580597    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"]
Nov 01 20:33:21.582450 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.582423    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-ngrfh"]
Nov 01 20:33:21.960319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:21.960287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:22.481584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.481545    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"]
Nov 01 20:33:22.494122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.494077    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"]
Nov 01 20:33:22.544165 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.544133    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:22.631960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.631920    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:33:22.644000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.643969    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:22.712245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.712207    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" path="/var/lib/kubelet/pods/6b718eed-0b7f-48f0-9a27-a03bbd79850c/volumes"
Nov 01 20:33:22.750969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.750876    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:22.845458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:22.845423    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:23.057013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:23.056923    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:23.153362 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:23.153320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:23.752961 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:23.752917    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:23.853513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:23.853479    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:24.361369 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:24.361303    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:25.055323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:25.055280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:25.650970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:25.650906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:26.526855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:26.526803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:27.051616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:27.051564    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:27.482575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:27.482532    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerName="payload-processing" containerID="cri-o://914d8227cc24e471de9b2c2e6e1b0bf48a891aac83435950d9cb4c9021fbc6cf" gracePeriod=25
Nov 01 20:33:27.494934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:27.494895    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerName="payload-pre-processing" containerID="cri-o://a4ce211bc6631bfec98c8875d883c2644fe70ad2ef5f7e1c83220b075297fd49" gracePeriod=25
Nov 01 20:33:27.677545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:27.677508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:28.062531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.062492    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"]
Nov 01 20:33:28.063156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063127    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy"
Nov 01 20:33:28.063156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063154    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063174    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063182    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063192    6479 cpu_manager.go:414] "containerMap: removing container" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063200    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063214    6479 cpu_manager.go:414] "containerMap: removing container" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerName="payload-pre-processing"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063222    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerName="payload-pre-processing"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063245    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063253    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063266    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063274    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063288    6479 cpu_manager.go:414] "containerMap: removing container" podUID="27c6e266-02c6-4d4c-8b78-e4ba53a3751d" containerName="maas-api"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063296    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c6e266-02c6-4d4c-8b78-e4ba53a3751d" containerName="maas-api"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063311    6479 cpu_manager.go:414] "containerMap: removing container" podUID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" containerName="main"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063319    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" containerName="main"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063327    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063333    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063349    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063357    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063366    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063374    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063383    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6b63897e-3823-4d01-a306-c9e921bb82fa" containerName="payload-pre-processing"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063396    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b63897e-3823-4d01-a306-c9e921bb82fa" containerName="payload-pre-processing"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063406    6479 cpu_manager.go:414] "containerMap: removing container" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063410    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063419    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy"
Nov 01 20:33:28.063407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063423    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063430    6479 cpu_manager.go:414] "containerMap: removing container" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063435    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063445    6479 cpu_manager.go:414] "containerMap: removing container" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063451    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063460    6479 cpu_manager.go:414] "containerMap: removing container" podUID="78512e90-9f27-4cee-8655-c4bcb41ea1ea" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063465    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78512e90-9f27-4cee-8655-c4bcb41ea1ea" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063471    6479 cpu_manager.go:414] "containerMap: removing container" podUID="cf650dd7-0480-4237-9ac5-2267acad84d5" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063475    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf650dd7-0480-4237-9ac5-2267acad84d5" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063483    6479 cpu_manager.go:414] "containerMap: removing container" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063487    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063498    6479 cpu_manager.go:414] "containerMap: removing container" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063502    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063508    6479 cpu_manager.go:414] "containerMap: removing container" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063513    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063520    6479 cpu_manager.go:414] "containerMap: removing container" podUID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063526    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063535    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063544    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063551    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d3644b62-073a-4e34-acc2-b0657b254966" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063556    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3644b62-073a-4e34-acc2-b0657b254966" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063563    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063567    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063576    6479 cpu_manager.go:414] "containerMap: removing container" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" containerName="test-curl-7426-ee9a91"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063580    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" containerName="test-curl-7426-ee9a91"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063591    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e39fa649-e41d-4ed0-9858-5024d1fc3a5b" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063596    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39fa649-e41d-4ed0-9858-5024d1fc3a5b" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063602    6479 cpu_manager.go:414] "containerMap: removing container" podUID="e20d8956-a771-4355-ac23-c47ff178585b" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063607    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20d8956-a771-4355-ac23-c47ff178585b" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063616    6479 cpu_manager.go:414] "containerMap: removing container" podUID="462fa0ea-b9e2-4515-89b6-bc93de691a95" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063621    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462fa0ea-b9e2-4515-89b6-bc93de691a95" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063629    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063633    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063661    6479 cpu_manager.go:414] "containerMap: removing container" podUID="9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063665    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063741    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="165c30dd-50e2-4a11-a761-29fddb64c414" containerName="test-curl-7426-ee9a91"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063752    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39fa649-e41d-4ed0-9858-5024d1fc3a5b" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063759    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20d8956-a771-4355-ac23-c47ff178585b" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063765    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="462fa0ea-b9e2-4515-89b6-bc93de691a95" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063772    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54274d3-3b0d-434f-8da7-6457847570fc" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063778    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f79e45c-d324-43ef-a7cd-efedb4fd9a8d" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063784    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4615003-28af-41a2-a79e-416ce58f2be7" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063789    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063794    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063799    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f68aa6-d7bc-4230-b5ea-6bcf505a7f53" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063804    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b718eed-0b7f-48f0-9a27-a03bbd79850c" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063808    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063813    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c6e266-02c6-4d4c-8b78-e4ba53a3751d" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063819    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f7f2ba-8bf9-470e-b738-000dcb16f3c4" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063826    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063833    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfa88dc-de2b-4918-80f9-7f1ad6dbd4c8" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063838    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b63897e-3823-4d01-a306-c9e921bb82fa" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063843    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="07174d89-b04b-47e6-8fd8-e8cfa611c75a" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063849    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af32888-a7a6-444e-be15-22035c3373ac" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063853    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff981dbc-864a-40a9-8908-df562cae4c22" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063860    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d665fd-290f-4d5f-8b56-ad7bbaebeaf9" containerName="istio-proxy"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063865    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="78512e90-9f27-4cee-8655-c4bcb41ea1ea" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063870    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf650dd7-0480-4237-9ac5-2267acad84d5" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063875    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cce8947-6a76-450d-b0d9-38d96efe2a0c" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063882    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f5dfca-b212-43a2-b3ba-4e316cdbbd71" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063887    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2f7d92-05fc-4a3d-95ba-beea9241bab6" containerName="payload-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063892    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="824bbcf5-e5e1-45a4-958a-bedb8740ad3e" containerName="maas-api"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063897    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c852ae-b370-4114-bcb1-1c5fd54e8328" containerName="main"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063902    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3644b62-073a-4e34-acc2-b0657b254966" containerName="payload-pre-processing"
Nov 01 20:33:28.065142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.063908    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f206db-9d47-4687-9cdc-884371b94797" containerName="istio-proxy"
Nov 01 20:33:28.068544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.068524    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.077752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.077593    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"]
Nov 01 20:33:28.096810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.096765    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbbx\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.096994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.096819    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.096994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.096960    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.097101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.096995    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.097138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.097088    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.097138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.097130    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.097216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.097200    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.097256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.097228    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.097367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.097350    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198462    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198578    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbbx\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198610    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198699    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198727    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198767    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.198803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198790    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198820    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.198845    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.199048    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.199126    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.199288    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199464 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.199438    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.199627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.199591    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.201272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.201247    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.201588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.201563    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.206091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.206055    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.206261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.206210    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbbx\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.253308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.253272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:28.387908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.387824    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:28.670686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.670633    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"]
Nov 01 20:33:28.673977 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:33:28.673938    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189aa725_bf08_422a_9e29_1b2c96780c2f.slice/crio-0d4d0c55cb5e7e7e68025297c77a4f2cca05b1d250f542c53555bf616d474bbb WatchSource:0}: Error finding container 0d4d0c55cb5e7e7e68025297c77a4f2cca05b1d250f542c53555bf616d474bbb: Status 404 returned error can't find the container with id 0d4d0c55cb5e7e7e68025297c77a4f2cca05b1d250f542c53555bf616d474bbb
Nov 01 20:33:28.676420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.676391    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:33:28.676526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.676465    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:33:28.676526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.676498    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:33:28.750551 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.750521    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:33:28.920529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:28.920427    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:29.142034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.141728    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:29.193784 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:29.193688    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:33:29.239275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.239241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:29.594119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.594074    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:29.594344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.594086    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" podUID="eede4389-f540-4c70-832e-a55e6198a442" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:29.609296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.609262    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" event={"ID":"189aa725-bf08-422a-9e29-1b2c96780c2f","Type":"ContainerStarted","Data":"a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1"}
Nov 01 20:33:29.609460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.609314    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" event={"ID":"189aa725-bf08-422a-9e29-1b2c96780c2f","Type":"ContainerStarted","Data":"0d4d0c55cb5e7e7e68025297c77a4f2cca05b1d250f542c53555bf616d474bbb"}
Nov 01 20:33:29.632349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.632291    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podStartSLOduration=1.63227608 podStartE2EDuration="1.63227608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:29.632061782 +0000 UTC m=+2055.047694281" watchObservedRunningTime="2026-11-01 20:33:29.63227608 +0000 UTC m=+2055.047908644"
Nov 01 20:33:29.720298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.720262    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:29.819613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:29.819573    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:30.076737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:30.074242    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:30.388995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:30.388912    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:30.390476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:30.390443    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:33:30.390615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:30.390491    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 01 20:33:30.476239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:30.476202    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:30.828843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:30.828802    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:31.077511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:31.077433    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:31.389241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:31.389202    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:33:31.389440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:31.389270    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 01 20:33:31.663261 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:31.663166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:32.367670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:32.357087    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:32.388967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:32.388923    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:33:32.389140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:32.388990    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 01 20:33:32.962672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:32.951625    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:33.283002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.282883    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:33.389117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.389066    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:33:33.389673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.389138    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 01 20:33:33.434234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.434188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:33.534224 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.534140    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:33.647783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.647728    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:33.990534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:33.990493    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:33:34.071676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.070097    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:33:34.088741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.088705    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume\") pod \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") "
Nov 01 20:33:34.088950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.088777    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg\" (UniqueName: \"kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg\") pod \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\" (UID: \"51356aec-ac1a-46a0-8eb2-cb56aa4e276a\") "
Nov 01 20:33:34.089382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.089354    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume" pod "51356aec-ac1a-46a0-8eb2-cb56aa4e276a" (UID: "51356aec-ac1a-46a0-8eb2-cb56aa4e276a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:34.092521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.092496    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg" pod "51356aec-ac1a-46a0-8eb2-cb56aa4e276a" (UID: "51356aec-ac1a-46a0-8eb2-cb56aa4e276a"). InnerVolumeSpecName "kube-api-access-z5mrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:34.190348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.190307    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj\" (UniqueName: \"kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj\") pod \"eede4389-f540-4c70-832e-a55e6198a442\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") "
Nov 01 20:33:34.190534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.190387    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume\") pod \"eede4389-f540-4c70-832e-a55e6198a442\" (UID: \"eede4389-f540-4c70-832e-a55e6198a442\") "
Nov 01 20:33:34.190932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.190677    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:34.190932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.190702    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mrg\" (UniqueName: \"kubernetes.io/projected/51356aec-ac1a-46a0-8eb2-cb56aa4e276a-kube-api-access-z5mrg\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:34.191120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.191065    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume" pod "eede4389-f540-4c70-832e-a55e6198a442" (UID: "eede4389-f540-4c70-832e-a55e6198a442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:34.192971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.192947    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj" pod "eede4389-f540-4c70-832e-a55e6198a442" (UID: "eede4389-f540-4c70-832e-a55e6198a442"). InnerVolumeSpecName "kube-api-access-c8tcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:34.278692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.278486    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:34.291494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.291410    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8tcj\" (UniqueName: \"kubernetes.io/projected/eede4389-f540-4c70-832e-a55e6198a442-kube-api-access-c8tcj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:34.291494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.291441    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eede4389-f540-4c70-832e-a55e6198a442-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:34.388428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.388383    6479 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:33:34.388611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.388467    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 01 20:33:34.435167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.435112    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:34.654905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.654808    6479 generic.go:356] "Generic (PLEG): container finished" podID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerID="3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c" exitCode=0
Nov 01 20:33:34.654905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.654853    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"
Nov 01 20:33:34.654905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.654893    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" event={"ID":"51356aec-ac1a-46a0-8eb2-cb56aa4e276a","Type":"ContainerDied","Data":"3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c"}
Nov 01 20:33:34.655288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.654954    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq" event={"ID":"51356aec-ac1a-46a0-8eb2-cb56aa4e276a","Type":"ContainerDied","Data":"c6ed8d2a2df3aad1c234a73cef449846bad8f71aa0db4cb6162ef407a1820f16"}
Nov 01 20:33:34.655288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.654984    6479 scope.go:122] "RemoveContainer" containerID="3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c"
Nov 01 20:33:34.656499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.656474    6479 generic.go:356] "Generic (PLEG): container finished" podID="eede4389-f540-4c70-832e-a55e6198a442" containerID="f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771" exitCode=0
Nov 01 20:33:34.656626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.656509    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" event={"ID":"eede4389-f540-4c70-832e-a55e6198a442","Type":"ContainerDied","Data":"f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771"}
Nov 01 20:33:34.656626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.656519    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"
Nov 01 20:33:34.656626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.656546    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5" event={"ID":"eede4389-f540-4c70-832e-a55e6198a442","Type":"ContainerDied","Data":"1e2c9e804c904d420211ccd8e57cd1affaf6905d849e03140df3d094eb0daaba"}
Nov 01 20:33:34.665062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.665038    6479 scope.go:122] "RemoveContainer" containerID="3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c"
Nov 01 20:33:34.665416 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:34.665390    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c\": container with ID starting with 3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c not found: ID does not exist" containerID="3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c"
Nov 01 20:33:34.665519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.665427    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c"} err="failed to get container status \"3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c\": rpc error: code = NotFound desc = could not find container \"3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c\": container with ID starting with 3fc41ae2d3c50ab57bf8f5b68a7d892f7c96d14ad0c74fda0362c6cfdd622b8c not found: ID does not exist"
Nov 01 20:33:34.665519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.665449    6479 scope.go:122] "RemoveContainer" containerID="f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771"
Nov 01 20:33:34.677855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.677824    6479 scope.go:122] "RemoveContainer" containerID="f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771"
Nov 01 20:33:34.677966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.677834    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"]
Nov 01 20:33:34.678226 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:34.678201    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771\": container with ID starting with f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771 not found: ID does not exist" containerID="f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771"
Nov 01 20:33:34.678296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.678235    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771"} err="failed to get container status \"f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771\": rpc error: code = NotFound desc = could not find container \"f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771\": container with ID starting with f5ad00ca3269937c36bab5a7ffd5b0c0c86dfc63aecc16cbfc005f64e788c771 not found: ID does not exist"
Nov 01 20:33:34.682809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.682773    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-9be196-7cdb7588f5-lpfpq"]
Nov 01 20:33:34.695246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.695177    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"]
Nov 01 20:33:34.697259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.697230    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9be196-6ddd5cdc-nsfp5"]
Nov 01 20:33:34.715030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.714994    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" path="/var/lib/kubelet/pods/51356aec-ac1a-46a0-8eb2-cb56aa4e276a/volumes"
Nov 01 20:33:34.715573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:34.715553    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eede4389-f540-4c70-832e-a55e6198a442" path="/var/lib/kubelet/pods/eede4389-f540-4c70-832e-a55e6198a442/volumes"
Nov 01 20:33:35.125104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.125072    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:35.398469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.396208    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:35.398469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.396743    6479 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" secret="" err="secret \"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\" not found"
Nov 01 20:33:35.398469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.397702    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:35.402669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.399029    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:35.424413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.424375    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"]
Nov 01 20:33:35.424689 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.424626    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy" containerID="cri-o://956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831" gracePeriod=30
Nov 01 20:33:35.427489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.427465    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"]
Nov 01 20:33:35.455516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.455485    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:35.536250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.536207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:35.667345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.667231    6479 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" secret="" err="secret \"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\" not found"
Nov 01 20:33:35.919998 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:35.919914    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:36.342155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:36.342120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:36.672290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:36.672245    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy" containerID="cri-o://a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1" gracePeriod=30
Nov 01 20:33:36.957466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:36.957433    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:37.645031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:37.644996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:38.030354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:38.030310    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"]
Nov 01 20:33:38.042775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:38.042738    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"]
Nov 01 20:33:38.127960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:38.127926    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:38.228079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:38.228041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:38.360124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:38.360045    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:38.678410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:38.677263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:39.126614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:39.126581    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:39.385958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:39.385869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:40.159424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.159377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:40.694553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.694523    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:33:40.695294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.695263    6479 generic.go:356] "Generic (PLEG): container finished" podID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerID="956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831" exitCode=0
Nov 01 20:33:40.695403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.695328    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" event={"ID":"d41b185c-14cd-4e06-9e7c-63fd65f437ca","Type":"ContainerDied","Data":"956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831"}
Nov 01 20:33:40.695403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.695385    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq" event={"ID":"d41b185c-14cd-4e06-9e7c-63fd65f437ca","Type":"ContainerDied","Data":"890deec4b6cd3891a793cfcea3f144c33b624de2d80d5de93dfb8770dcdd0963"}
Nov 01 20:33:40.695505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.695412    6479 scope.go:122] "RemoveContainer" containerID="956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831"
Nov 01 20:33:40.708284 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.708257    6479 scope.go:122] "RemoveContainer" containerID="956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831"
Nov 01 20:33:40.708627 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:40.708593    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831\": container with ID starting with 956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831 not found: ID does not exist" containerID="956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831"
Nov 01 20:33:40.708766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.708636    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831"} err="failed to get container status \"956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831\": rpc error: code = NotFound desc = could not find container \"956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831\": container with ID starting with 956c602216a72d68341702c12cd487e91a9ce72625396297a542a273e52c1831 not found: ID does not exist"
Nov 01 20:33:40.747055 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.747011    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:33:40.803691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.803631    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.803901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.803705    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.803901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.803742    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.803901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.803792    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.803901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.803890    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.804393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804107    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.804516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804156    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.804516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804266    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.804516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804483    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.804744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804721    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.804818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804785    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert\") pod \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\" (UID: \"d41b185c-14cd-4e06-9e7c-63fd65f437ca\") "
Nov 01 20:33:40.804876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.804826    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.805249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.805224    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:40.805605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.805584    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.805718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.805612    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.805718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.805627    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.805718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.805641    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.806005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.805977    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.806366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.806345    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:40.806453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.806403    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:40.806566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.806550    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "kube-api-access-6x7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:40.806619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.806583    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy" pod "d41b185c-14cd-4e06-9e7c-63fd65f437ca" (UID: "d41b185c-14cd-4e06-9e7c-63fd65f437ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.831157 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.831116    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:40.907026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.906985    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.907026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.907015    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x7lq\" (UniqueName: \"kubernetes.io/projected/d41b185c-14cd-4e06-9e7c-63fd65f437ca-kube-api-access-6x7lq\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.907026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.907028    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.907026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.907037    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.907304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:40.907046    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41b185c-14cd-4e06-9e7c-63fd65f437ca-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:41.443348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.443313    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:41.702967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.702938    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"
Nov 01 20:33:41.762176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.762139    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"]
Nov 01 20:33:41.769840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.769812    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6xdbq"]
Nov 01 20:33:41.906145 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.906109    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:41.931430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.931396    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:41.994023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:41.993928    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:42.120207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120165    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.120395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120230    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.120395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120283    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.120395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120311    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.120395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120345    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.120395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120375    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120408    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120563    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120611    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy\") pod \"189aa725-bf08-422a-9e29-1b2c96780c2f\" (UID: \"189aa725-bf08-422a-9e29-1b2c96780c2f\") "
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120703    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120703    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120942    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.120957    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.121028    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.121063    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.121363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.121314    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:42.123253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.123156    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "kube-api-access-gpbbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:42.123253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.123228    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:42.123397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.123233    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:42.123397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.123362    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo" pod "189aa725-bf08-422a-9e29-1b2c96780c2f" (UID: "189aa725-bf08-422a-9e29-1b2c96780c2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:42.147385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.147192    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:42.221933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.221881    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.221933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.221929    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.222167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.221947    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.222167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.221962    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbbx\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-kube-api-access-gpbbx\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.222167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.221977    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/189aa725-bf08-422a-9e29-1b2c96780c2f-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.222167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.221990    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/189aa725-bf08-422a-9e29-1b2c96780c2f-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.222167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.222004    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/189aa725-bf08-422a-9e29-1b2c96780c2f-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:42.707972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.707696    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" path="/var/lib/kubelet/pods/d41b185c-14cd-4e06-9e7c-63fd65f437ca/volumes"
Nov 01 20:33:42.709109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.709080    6479 generic.go:356] "Generic (PLEG): container finished" podID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerID="a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1" exitCode=0
Nov 01 20:33:42.709220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.709121    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"
Nov 01 20:33:42.709220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.709152    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" event={"ID":"189aa725-bf08-422a-9e29-1b2c96780c2f","Type":"ContainerDied","Data":"a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1"}
Nov 01 20:33:42.709220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.709211    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c" event={"ID":"189aa725-bf08-422a-9e29-1b2c96780c2f","Type":"ContainerDied","Data":"0d4d0c55cb5e7e7e68025297c77a4f2cca05b1d250f542c53555bf616d474bbb"}
Nov 01 20:33:42.709357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.709242    6479 scope.go:122] "RemoveContainer" containerID="a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1"
Nov 01 20:33:42.719695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.719639    6479 scope.go:122] "RemoveContainer" containerID="a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1"
Nov 01 20:33:42.720616 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:42.720574    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1\": container with ID starting with a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1 not found: ID does not exist" containerID="a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1"
Nov 01 20:33:42.721085 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.720667    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1"} err="failed to get container status \"a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1\": rpc error: code = NotFound desc = could not find container \"a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1\": container with ID starting with a125278cd574ab8ef868d0b9fb1598f568773382536af9b35d83806126ff2fd1 not found: ID does not exist"
Nov 01 20:33:42.735790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.735698    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"]
Nov 01 20:33:42.742442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.742415    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-6jv6c"]
Nov 01 20:33:42.761712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:42.761678    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:43.032794 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:43.032018    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerName="payload-processing" containerID="cri-o://278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3" gracePeriod=25
Nov 01 20:33:43.263583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:43.263535    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:43.369733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:43.366915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:43.684902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:43.684784    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" containerName="payload-pre-processing" containerID="cri-o://890fb8114b87f9dc6805c7269712868e7ec7be40feeee8bf0c7bf57a9d51e5c9" gracePeriod=25
Nov 01 20:33:43.959451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:43.959402    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:44.059564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.059523    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:44.159195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.159157    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:44.272932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.272836    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:44.348657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.348603    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:44.348838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.348701    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:44.442612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.442580    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:44.707662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.707605    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" path="/var/lib/kubelet/pods/189aa725-bf08-422a-9e29-1b2c96780c2f/volumes"
Nov 01 20:33:44.799726 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.799695    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:44.900204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:44.900168    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:45.452659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:45.452603    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:45.551025 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:45.550991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:46.129318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:46.129279    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:46.750238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:46.750204    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:47.239034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.238998    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:47.343341 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.343292    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:47.554633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.554597    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:47.738783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.738750    6479 generic.go:356] "Generic (PLEG): container finished" podID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerID="914d8227cc24e471de9b2c2e6e1b0bf48a891aac83435950d9cb4c9021fbc6cf" exitCode=0
Nov 01 20:33:47.738929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.738829    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" event={"ID":"8431cbc4-bd80-4332-a51d-155e91edb9da","Type":"ContainerDied","Data":"914d8227cc24e471de9b2c2e6e1b0bf48a891aac83435950d9cb4c9021fbc6cf"}
Nov 01 20:33:47.738929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.738896    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w" event={"ID":"8431cbc4-bd80-4332-a51d-155e91edb9da","Type":"ContainerDied","Data":"19484c2f71ac4cae5152653fbf81e0f197ecf5d51592de59767b0f64e5e7d3b8"}
Nov 01 20:33:47.738929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.738923    6479 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19484c2f71ac4cae5152653fbf81e0f197ecf5d51592de59767b0f64e5e7d3b8"
Nov 01 20:33:47.740938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.740908    6479 generic.go:356] "Generic (PLEG): container finished" podID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerID="a4ce211bc6631bfec98c8875d883c2644fe70ad2ef5f7e1c83220b075297fd49" exitCode=0
Nov 01 20:33:47.741074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.740956    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" event={"ID":"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da","Type":"ContainerDied","Data":"a4ce211bc6631bfec98c8875d883c2644fe70ad2ef5f7e1c83220b075297fd49"}
Nov 01 20:33:47.747731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.747670    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:33:47.775212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.775181    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7\" (UniqueName: \"kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7\") pod \"8431cbc4-bd80-4332-a51d-155e91edb9da\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") "
Nov 01 20:33:47.775697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.775245    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume\") pod \"8431cbc4-bd80-4332-a51d-155e91edb9da\" (UID: \"8431cbc4-bd80-4332-a51d-155e91edb9da\") "
Nov 01 20:33:47.775697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.775612    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume" pod "8431cbc4-bd80-4332-a51d-155e91edb9da" (UID: "8431cbc4-bd80-4332-a51d-155e91edb9da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:47.777315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.777285    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7" pod "8431cbc4-bd80-4332-a51d-155e91edb9da" (UID: "8431cbc4-bd80-4332-a51d-155e91edb9da"). InnerVolumeSpecName "kube-api-access-gctp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:47.792659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.792621    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:33:47.876417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.876377    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g\" (UniqueName: \"kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g\") pod \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") "
Nov 01 20:33:47.876576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.876453    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume\") pod \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\" (UID: \"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da\") "
Nov 01 20:33:47.876745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.876730    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctp7\" (UniqueName: \"kubernetes.io/projected/8431cbc4-bd80-4332-a51d-155e91edb9da-kube-api-access-gctp7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:47.876795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.876748    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8431cbc4-bd80-4332-a51d-155e91edb9da-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:47.876912 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.876886    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume" pod "0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" (UID: "0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:47.878599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.878577    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g" pod "0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" (UID: "0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da"). InnerVolumeSpecName "kube-api-access-7rn8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:47.978056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.978017    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:47.978056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:47.978049    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rn8g\" (UniqueName: \"kubernetes.io/projected/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da-kube-api-access-7rn8g\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:48.155038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.154943    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:48.746954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.746923    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"
Nov 01 20:33:48.746954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.746948    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8" event={"ID":"0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da","Type":"ContainerDied","Data":"8d1fd364fc84026b1a6f1e1589fa93d7f5ba5f5b2f6743ce4c1a3909de028a88"}
Nov 01 20:33:48.747200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.747008    6479 scope.go:122] "RemoveContainer" containerID="a4ce211bc6631bfec98c8875d883c2644fe70ad2ef5f7e1c83220b075297fd49"
Nov 01 20:33:48.747254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.747234    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"
Nov 01 20:33:48.762693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.762660    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"]
Nov 01 20:33:48.765847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.765808    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:48.767105 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.767085    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e83b4b-557c8556cc-f742w"]
Nov 01 20:33:48.776744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.776719    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"]
Nov 01 20:33:48.780129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:48.780094    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e83b4b-c7f9d755-nf2j8"]
Nov 01 20:33:49.451638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:49.451608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:50.069515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:50.069470    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:50.671212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:50.671178    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:50.705559 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:50.705526    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" path="/var/lib/kubelet/pods/0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da/volumes"
Nov 01 20:33:50.705905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:50.705890    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" path="/var/lib/kubelet/pods/8431cbc4-bd80-4332-a51d-155e91edb9da/volumes"
Nov 01 20:33:51.376399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:51.376341    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:51.456132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:51.456095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:51.553915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:51.553871    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:51.940532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:51.940499    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:52.059662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:52.059593    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:53.341500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:53.341462    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:33:54.348950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:54.348912    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:54.349346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:54.348980    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:57.550960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:57.550878    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:57.662720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:57.662682    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:57.907374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:57.907278    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:58.245247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:33:58.245213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:33:59.170249 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:33:59.170214    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:34:00.090784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:00.090751    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-59thg_6f62c32d-0014-4bf3-853c-dd0cfc0f6b66/manager/0.log"
Nov 01 20:34:00.332332 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:00.332301    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v5k66_8c6137ad-fb29-4667-be40-173018e00d94/limitador/0.log"
Nov 01 20:34:01.948061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:01.948032    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-trh22_eff10696-0cfb-4433-ad91-3d0e817fd274/manager/0.log"
Nov 01 20:34:03.288428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.288396    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:34:03.329361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.329262    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume\") pod \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") "
Nov 01 20:34:03.329361 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.329343    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm\" (UniqueName: \"kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm\") pod \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\" (UID: \"1359fa9e-f0ad-4b34-8c98-4361fef33f7f\") "
Nov 01 20:34:03.329762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.329731    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume" pod "1359fa9e-f0ad-4b34-8c98-4361fef33f7f" (UID: "1359fa9e-f0ad-4b34-8c98-4361fef33f7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:03.331490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.331467    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm" pod "1359fa9e-f0ad-4b34-8c98-4361fef33f7f" (UID: "1359fa9e-f0ad-4b34-8c98-4361fef33f7f"). InnerVolumeSpecName "kube-api-access-fmlwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:03.430268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.430227    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.430268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.430261    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmlwm\" (UniqueName: \"kubernetes.io/projected/1359fa9e-f0ad-4b34-8c98-4361fef33f7f-kube-api-access-fmlwm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.823113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.823071    6479 generic.go:356] "Generic (PLEG): container finished" podID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerID="278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3" exitCode=0
Nov 01 20:34:03.823287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.823145    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"
Nov 01 20:34:03.823287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.823157    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" event={"ID":"1359fa9e-f0ad-4b34-8c98-4361fef33f7f","Type":"ContainerDied","Data":"278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3"}
Nov 01 20:34:03.823287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.823211    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7" event={"ID":"1359fa9e-f0ad-4b34-8c98-4361fef33f7f","Type":"ContainerDied","Data":"687c48cf9c55612434c699d30cfee2a3591a59c38571f0a6417dfe11da89cbb6"}
Nov 01 20:34:03.823287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.823234    6479 scope.go:122] "RemoveContainer" containerID="278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3"
Nov 01 20:34:03.824869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.824844    6479 generic.go:356] "Generic (PLEG): container finished" podID="dc9cd2de-8613-46c3-8239-8853debf1530" containerID="890fb8114b87f9dc6805c7269712868e7ec7be40feeee8bf0c7bf57a9d51e5c9" exitCode=0
Nov 01 20:34:03.824980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.824879    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" event={"ID":"dc9cd2de-8613-46c3-8239-8853debf1530","Type":"ContainerDied","Data":"890fb8114b87f9dc6805c7269712868e7ec7be40feeee8bf0c7bf57a9d51e5c9"}
Nov 01 20:34:03.834574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.834550    6479 scope.go:122] "RemoveContainer" containerID="278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3"
Nov 01 20:34:03.834892 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:34:03.834871    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3\": container with ID starting with 278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3 not found: ID does not exist" containerID="278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3"
Nov 01 20:34:03.834942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.834901    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3"} err="failed to get container status \"278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3\": rpc error: code = NotFound desc = could not find container \"278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3\": container with ID starting with 278c5d7b195c9c09f76e9eb2eee36f24c02f4acb2763821b3fbd5e422fd580c3 not found: ID does not exist"
Nov 01 20:34:03.845718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.845691    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"]
Nov 01 20:34:03.847530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.847504    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e38ef4-64c649c7fc-p5cg7"]
Nov 01 20:34:03.925351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:03.925328    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:34:04.036053 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.036020    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth\" (UniqueName: \"kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth\") pod \"dc9cd2de-8613-46c3-8239-8853debf1530\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") "
Nov 01 20:34:04.036227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.036064    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume\") pod \"dc9cd2de-8613-46c3-8239-8853debf1530\" (UID: \"dc9cd2de-8613-46c3-8239-8853debf1530\") "
Nov 01 20:34:04.036509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.036484    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume" pod "dc9cd2de-8613-46c3-8239-8853debf1530" (UID: "dc9cd2de-8613-46c3-8239-8853debf1530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:04.038291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.038262    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth" pod "dc9cd2de-8613-46c3-8239-8853debf1530" (UID: "dc9cd2de-8613-46c3-8239-8853debf1530"). InnerVolumeSpecName "kube-api-access-8wqth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:04.136837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.136800    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqth\" (UniqueName: \"kubernetes.io/projected/dc9cd2de-8613-46c3-8239-8853debf1530-kube-api-access-8wqth\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:04.136837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.136834    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9cd2de-8613-46c3-8239-8853debf1530-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:04.706197 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.706158    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" path="/var/lib/kubelet/pods/1359fa9e-f0ad-4b34-8c98-4361fef33f7f/volumes"
Nov 01 20:34:04.832835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.832806    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"
Nov 01 20:34:04.832983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.832808    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg" event={"ID":"dc9cd2de-8613-46c3-8239-8853debf1530","Type":"ContainerDied","Data":"05c2b5780fd9c86869105adb745120a91f1bdedcf1cf6d72edc1652f2ec53496"}
Nov 01 20:34:04.832983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.832932    6479 scope.go:122] "RemoveContainer" containerID="890fb8114b87f9dc6805c7269712868e7ec7be40feeee8bf0c7bf57a9d51e5c9"
Nov 01 20:34:04.848867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.848837    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"]
Nov 01 20:34:04.852408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:04.852379    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e38ef4-66d56c5bcf-ng8cg"]
Nov 01 20:34:06.705853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:06.705817    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" path="/var/lib/kubelet/pods/dc9cd2de-8613-46c3-8239-8853debf1530/volumes"
Nov 01 20:34:06.848206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:06.848162    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:07.424280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.424247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:07.796459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.796419    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"]
Nov 01 20:34:07.797036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797015    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797038    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797054    6479 cpu_manager.go:414] "containerMap: removing container" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797061    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797078    6479 cpu_manager.go:414] "containerMap: removing container" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" containerName="payload-pre-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797086    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" containerName="payload-pre-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797096    6479 cpu_manager.go:414] "containerMap: removing container" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerName="payload-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797105    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerName="payload-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797120    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerName="payload-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797128    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerName="payload-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797153    6479 cpu_manager.go:414] "containerMap: removing container" podUID="eede4389-f540-4c70-832e-a55e6198a442" containerName="payload-pre-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797160    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eede4389-f540-4c70-832e-a55e6198a442" containerName="payload-pre-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797175    6479 cpu_manager.go:414] "containerMap: removing container" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerName="payload-pre-processing"
Nov 01 20:34:07.797186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797181    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerName="payload-pre-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797205    6479 cpu_manager.go:414] "containerMap: removing container" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797213    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797225    6479 cpu_manager.go:414] "containerMap: removing container" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerName="payload-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797232    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerName="payload-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797335    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41b185c-14cd-4e06-9e7c-63fd65f437ca" containerName="istio-proxy"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797350    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="189aa725-bf08-422a-9e29-1b2c96780c2f" containerName="istio-proxy"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797359    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9cd2de-8613-46c3-8239-8853debf1530" containerName="payload-pre-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797369    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="51356aec-ac1a-46a0-8eb2-cb56aa4e276a" containerName="payload-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797379    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8431cbc4-bd80-4332-a51d-155e91edb9da" containerName="payload-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797388    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="eede4389-f540-4c70-832e-a55e6198a442" containerName="payload-pre-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797397    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a98fd7b-5575-4f8b-8a1e-4dda94d8c7da" containerName="payload-pre-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797408    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6263de7-73f1-434c-bf75-230cc71de2f8" containerName="istio-proxy"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797423    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="1359fa9e-f0ad-4b34-8c98-4361fef33f7f" containerName="payload-processing"
Nov 01 20:34:07.797908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.797434    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c65fc48-3041-4a60-a9da-b7ed8ab4d83c" containerName="istio-proxy"
Nov 01 20:34:07.800728 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.800707    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.810212 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.810188    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-1bfa0e-openshift-default-dockercfg-hbng5\""
Nov 01 20:34:07.811598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.811568    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"]
Nov 01 20:34:07.839688 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.839633    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:07.876401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876362    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876416    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876521    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876581 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876545    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876614    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876695    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mmn\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876721    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876746    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.876856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.876785    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.966702 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.966664    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:07.978279 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978246    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978298    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978405    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978440    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978488    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978536    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mmn\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978575    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978616    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978641    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.978910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.978774    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.979133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.979109    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.979214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.979147    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.979377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.979341    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.979530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.979505    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.980836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.980811    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.981185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.981162    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.986275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.986252    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:07.986373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:07.986352    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mmn\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:08.116282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.116178    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:08.394236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.394155    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"]
Nov 01 20:34:08.397957 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:34:08.397923    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a249be_eaaf_43d0_a740_e92466687b8e.slice/crio-1c6d41cefde9b5a83907bbffa62f88cb7c6f29c744ad82c3d7ad9c18113fb2ea WatchSource:0}: Error finding container 1c6d41cefde9b5a83907bbffa62f88cb7c6f29c744ad82c3d7ad9c18113fb2ea: Status 404 returned error can't find the container with id 1c6d41cefde9b5a83907bbffa62f88cb7c6f29c744ad82c3d7ad9c18113fb2ea
Nov 01 20:34:08.400295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.400258    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:34:08.400417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.400326    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:34:08.400417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.400354    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:34:08.855314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.855269    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" event={"ID":"71a249be-eaaf-43d0-a740-e92466687b8e","Type":"ContainerStarted","Data":"f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"}
Nov 01 20:34:08.855853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.855330    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" event={"ID":"71a249be-eaaf-43d0-a740-e92466687b8e","Type":"ContainerStarted","Data":"1c6d41cefde9b5a83907bbffa62f88cb7c6f29c744ad82c3d7ad9c18113fb2ea"}
Nov 01 20:34:08.874738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:08.874687    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podStartSLOduration=1.874668767 podStartE2EDuration="1.874668767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:08.872967861 +0000 UTC m=+2094.288600336" watchObservedRunningTime="2026-11-01 20:34:08.874668767 +0000 UTC m=+2094.290301233"
Nov 01 20:34:09.117202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:09.117097    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:09.122562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:09.122525    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:09.861064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:09.861026    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:09.862020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:09.861995    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:11.115159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:11.115120    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:11.214929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:11.214888    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:12.226049 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.226012    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"]
Nov 01 20:34:12.230182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.230158    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.242222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.242189    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-main-1bfa0e-serving-cert\""
Nov 01 20:34:12.244621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.244208    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"]
Nov 01 20:34:12.283513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.283478    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:12.322926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.322890    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.323114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.322936    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.323114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.322964    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.323114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.323105    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2xv\" (UniqueName: \"kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.351541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.351510    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"]
Nov 01 20:34:12.355251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.355220    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.362815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.362439    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-main-1bfa0e\""
Nov 01 20:34:12.364552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.364501    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"]
Nov 01 20:34:12.374917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.374893    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:12.389605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.389580    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"]
Nov 01 20:34:12.393141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.393125    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.401390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.401369    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-main-1bfa0e-dockercfg-zhglm\""
Nov 01 20:34:12.403505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.403478    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"]
Nov 01 20:34:12.424563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.424533    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9fh\" (UniqueName: \"kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh\") pod \"payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") " pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.424769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.424612    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.424769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.424707    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.424769 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:34:12.424743    6479 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-serving-cert: secret "maas-api-e2e-worker-main-1bfa0e-serving-cert" not found
Nov 01 20:34:12.424769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.424745    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.425038 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:34:12.424803    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls podName:6f4e8ca3-61ec-43e5-87a7-b38f21316b1e nodeName:}" failed. No retries permitted until 2026-11-01 20:34:12.924781862 +0000 UTC m=+2098.340414316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls") pod "maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" (UID: "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e") : secret "maas-api-e2e-worker-main-1bfa0e-serving-cert" not found
Nov 01 20:34:12.425038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.424825    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") " pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.425038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.424883    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2xv\" (UniqueName: \"kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.425344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.425324    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.427442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.427410    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.433532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.433503    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2xv\" (UniqueName: \"kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.488465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.488383    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:12.525706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.525632    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfdw\" (UniqueName: \"kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw\") pod \"payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.525892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.525784    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") " pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.525892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.525811    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.525892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.525878    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9fh\" (UniqueName: \"kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh\") pod \"payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") " pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.526548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.526523    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") " pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.534479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.534449    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9fh\" (UniqueName: \"kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh\") pod \"payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") " pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.627236 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.627190    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfdw\" (UniqueName: \"kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw\") pod \"payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.627435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.627280    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.628081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.628061    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.635368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.635338    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfdw\" (UniqueName: \"kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw\") pod \"payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.670308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.670276    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:12.709112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.709070    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:12.931493 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.931457    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.933829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.933807    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\") pod \"maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:12.952304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.952268    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"]
Nov 01 20:34:12.953077 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:34:12.953045    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae245b6_740c_48da_a939_291af7712863.slice/crio-9bc78d0a4667c7209091727d158d51f3a56b140c5d99d33a96bbe4e7daa46a19 WatchSource:0}: Error finding container 9bc78d0a4667c7209091727d158d51f3a56b140c5d99d33a96bbe4e7daa46a19: Status 404 returned error can't find the container with id 9bc78d0a4667c7209091727d158d51f3a56b140c5d99d33a96bbe4e7daa46a19
Nov 01 20:34:12.992365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:12.992335    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"]
Nov 01 20:34:12.996188 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:34:12.996162    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b91524_aa26_4e14_9e16_a390623fd08c.slice/crio-424db1481fac4a3699db089e1f363d89d29851e181437534628342cbbef27f99 WatchSource:0}: Error finding container 424db1481fac4a3699db089e1f363d89d29851e181437534628342cbbef27f99: Status 404 returned error can't find the container with id 424db1481fac4a3699db089e1f363d89d29851e181437534628342cbbef27f99
Nov 01 20:34:13.066295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.066259    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:13.153478 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.153440    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:13.432295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.432267    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"]
Nov 01 20:34:13.434392 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:34:13.434358    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e8ca3_61ec_43e5_87a7_b38f21316b1e.slice/crio-d6b46acc8c9891d7488982465f08d1682460f735cc9ae0a5a7bd3d6c84f32bee WatchSource:0}: Error finding container d6b46acc8c9891d7488982465f08d1682460f735cc9ae0a5a7bd3d6c84f32bee: Status 404 returned error can't find the container with id d6b46acc8c9891d7488982465f08d1682460f735cc9ae0a5a7bd3d6c84f32bee
Nov 01 20:34:13.435933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.435900    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:34:13.743725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.743681    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:13.883360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.883322    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl" event={"ID":"3ae245b6-740c-48da-a939-291af7712863","Type":"ContainerStarted","Data":"2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8"}
Nov 01 20:34:13.883575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.883399    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl" event={"ID":"3ae245b6-740c-48da-a939-291af7712863","Type":"ContainerStarted","Data":"9bc78d0a4667c7209091727d158d51f3a56b140c5d99d33a96bbe4e7daa46a19"}
Nov 01 20:34:13.883575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.883430    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:13.884607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.884579    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" event={"ID":"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e","Type":"ContainerStarted","Data":"d6b46acc8c9891d7488982465f08d1682460f735cc9ae0a5a7bd3d6c84f32bee"}
Nov 01 20:34:13.886167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.886141    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj" event={"ID":"12b91524-aa26-4e14-9e16-a390623fd08c","Type":"ContainerStarted","Data":"7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b"}
Nov 01 20:34:13.886321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.886180    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj" event={"ID":"12b91524-aa26-4e14-9e16-a390623fd08c","Type":"ContainerStarted","Data":"424db1481fac4a3699db089e1f363d89d29851e181437534628342cbbef27f99"}
Nov 01 20:34:13.886321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.886264    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:13.900549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.900508    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl" podStartSLOduration=1.9004932490000002 podStartE2EDuration="1.900493249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:13.89991861 +0000 UTC m=+2099.315551083" watchObservedRunningTime="2026-11-01 20:34:13.900493249 +0000 UTC m=+2099.316125722"
Nov 01 20:34:13.917187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:13.917124    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj" podStartSLOduration=1.917105863 podStartE2EDuration="1.917105863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:13.916574807 +0000 UTC m=+2099.332207278" watchObservedRunningTime="2026-11-01 20:34:13.917105863 +0000 UTC m=+2099.332738339"
Nov 01 20:34:14.413149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:14.413069    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:14.711989 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:14.711958    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:34:14.893132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:14.893085    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" event={"ID":"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e","Type":"ContainerStarted","Data":"8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b"}
Nov 01 20:34:14.893393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:14.893319    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:14.910359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:14.910308    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" podStartSLOduration=2.380721811 podStartE2EDuration="2.910294286s" totalImagesPullingTime="529.572475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:14.907972621 +0000 UTC m=+2100.323605094" watchObservedRunningTime="2026-11-01 20:34:14.910294286 +0000 UTC m=+2100.325926823"
Nov 01 20:34:15.144405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:15.144314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:15.752050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:15.752012    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:16.449252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:16.449216    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:17.048813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:17.048783    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:17.745141 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:17.745105    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:18.341357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:18.341320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:18.826363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:18.826326    6479 scope.go:122] "RemoveContainer" containerID="914d8227cc24e471de9b2c2e6e1b0bf48a891aac83435950d9cb4c9021fbc6cf"
Nov 01 20:34:18.948876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:18.948837    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:19.646841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:19.646811    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:20.251803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:20.251762    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:20.840515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:20.840478    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:20.911104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:20.910866    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:34:21.551272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:21.551237    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:22.281734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:22.281697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:22.871685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:22.870949    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:23.149480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:23.149398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:23.549067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:23.549033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:24.147789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:24.147738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:24.743109 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:24.743072    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:24.896219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:24.896189    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:34:24.896400 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:24.896259    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:34:25.340304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:25.340270    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:25.984918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:25.984878    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:26.084774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:26.084732    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:26.524939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:26.524902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:26.627631 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:26.627598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:26.728982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:26.727169    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:27.478283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:27.478247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:27.578099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:27.578061    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:28.252923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:28.252883    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:28.958915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:28.958842    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:29.046891 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:29.046855    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:34:29.174572 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:34:29.174532    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:34:29.778708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:29.778661    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:30.363890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:30.363852    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:31.055674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.055618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:31.659021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.658968    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:31.990345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.990314    6479 generic.go:356] "Generic (PLEG): container finished" podID="71a249be-eaaf-43d0-a740-e92466687b8e" containerID="f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789" exitCode=0
Nov 01 20:34:31.990524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.990383    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" event={"ID":"71a249be-eaaf-43d0-a740-e92466687b8e","Type":"ContainerDied","Data":"f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"}
Nov 01 20:34:31.990849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.990834    6479 scope.go:122] "RemoveContainer" containerID="f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"
Nov 01 20:34:31.992217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.992188    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:34:31.992334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.992259    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:34:31.992334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:31.992293    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:34:32.339094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:32.338997    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:32.969670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:32.967334    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:32.997095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:32.997054    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" event={"ID":"71a249be-eaaf-43d0-a740-e92466687b8e","Type":"ContainerStarted","Data":"b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727"}
Nov 01 20:34:33.117327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:33.117291    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:33.117524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:33.117491    6479 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:34:33.117578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:33.117539    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 20:34:33.565319 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:33.565266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:34.117588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:34.117551    6479 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:34:34.118117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:34.117612    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 20:34:34.234457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:34.234423    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:34.845587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:34.845549    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:35.117545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:35.117468    6479 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:34:35.117731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:35.117536    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 20:34:35.449607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:35.449571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:36.117444 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:36.117402    6479 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:34:36.117630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:36.117465    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 20:34:36.145638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:36.145606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:36.763870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:36.763833    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:37.117138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:37.117058    6479 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:34:37.117138 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:37.117124    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 20:34:37.361119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:37.361080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:37.959449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:37.959404    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:38.116558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:38.116518    6479 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:34:38.116771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:38.116589    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 20:34:38.555352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:38.555314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:39.121887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:39.121848    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:39.122162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:39.122140    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:39.122850 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:39.122834    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:34:39.163232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:39.163199    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:39.758175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:39.758136    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:40.517360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:40.517322    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:41.054337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:41.054301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:41.656422 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:41.656363    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:42.453488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:42.453437    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:43.054884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:43.054843    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:43.747762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:43.747725    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:44.068340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:44.068247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:44.447754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:44.447721    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:45.067135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:45.067090    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:45.730566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:45.730528    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:34:45.889401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:45.889367    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:46.653722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:46.653619    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:47.362132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:47.362096    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:48.063191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:48.063143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:48.757910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:48.757873    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:49.396550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:49.396509    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:49.972206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:49.972168    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:50.607719 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:50.607682    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:51.155736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:51.155693    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:51.747475 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:51.747442    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:52.354174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:52.354140    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:52.984931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:52.984896    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:53.539829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:53.539790    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:54.238835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:54.238798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:54.954879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:54.954839    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:55.544896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:55.544855    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:56.282248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:56.282214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:56.851630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:56.851586    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:57.448629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:57.448591    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:58.050196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:58.050165    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:58.727667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:58.727609    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:34:59.170169 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:34:59.170090    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:34:59.346908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:34:59.346868    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:00.045186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:00.045145    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:00.822338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:00.822299    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:01.435873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:01.435835    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:01.536093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:01.536058    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:35:02.354530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:02.354492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:03.154764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:03.154722    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:03.750342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:03.750305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:04.458910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:04.458869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:05.048278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:05.048243    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:05.657463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:05.657416    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:06.248134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:06.248067    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:06.847465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:06.847404    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:07.525918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:07.525881    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:08.059075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:08.059033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:08.732745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:08.732710    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:09.351066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:09.351034    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:09.949742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:09.949702    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:10.551829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:10.551793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:11.144970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:11.144931    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:11.747563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:11.747529    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:12.452330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:12.452297    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:13.055408 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:13.055361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:13.654501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:13.654467    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:14.254873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:14.254822    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:18.158672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:18.157764    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:35:18.158672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:18.157848    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:35:18.161776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:18.161737    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:35:18.162023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:18.161991    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:35:23.189345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:23.189307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:26.766486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:26.766406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:26.858137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:26.858097    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:29.171334 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:35:29.171301    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:35:37.939950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:37.939912    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:38.353669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:38.353601    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:38.959374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:38.959332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:39.653875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:39.653844    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:40.004870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.004835    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"]
Nov 01 20:35:40.008436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.008417    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.026566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.026530    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"]
Nov 01 20:35:40.141604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.141559    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.141604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.141607    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.141946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.141726    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkdt\" (UniqueName: \"kubernetes.io/projected/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-kube-api-access-fjkdt\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.141946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.141841    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.141946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.141889    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.141946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.141936    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.142180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.142092    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.142180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.142124    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.142180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.142175    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.242942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.242898    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.242942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.242943    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.242982    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkdt\" (UniqueName: \"kubernetes.io/projected/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-kube-api-access-fjkdt\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243025    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243070    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243122    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243237    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243267    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243307    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243620    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243684    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243807 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243764    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.243907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.243836    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.244084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.244061    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.245510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.245489    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.245709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.245690    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.260674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.260583    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.260873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.260848    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkdt\" (UniqueName: \"kubernetes.io/projected/4e8677a8-ebe4-4a37-9a4e-e43551a314ff-kube-api-access-fjkdt\") pod \"maas-default-gateway-openshift-default-fd4b99879-989zh\" (UID: \"4e8677a8-ebe4-4a37-9a4e-e43551a314ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.323225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.323181    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:40.355606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.355574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:40.606548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.606520    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"]
Nov 01 20:35:40.609228 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:35:40.609198    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8677a8_ebe4_4a37_9a4e_e43551a314ff.slice/crio-45decad66768620408a903bb86166d25f4ccbbc2152cf146bdcd3009ecd3aeeb WatchSource:0}: Error finding container 45decad66768620408a903bb86166d25f4ccbbc2152cf146bdcd3009ecd3aeeb: Status 404 returned error can't find the container with id 45decad66768620408a903bb86166d25f4ccbbc2152cf146bdcd3009ecd3aeeb
Nov 01 20:35:40.611405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.611374    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:35:40.611509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.611452    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:35:40.611509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:40.611499    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:35:41.015206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:41.015161    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:41.350711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:41.350619    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" event={"ID":"4e8677a8-ebe4-4a37-9a4e-e43551a314ff","Type":"ContainerStarted","Data":"ee3b30b49ccb84392dbe046961c165a0901a3f09e93ef3a09820a6028c605704"}
Nov 01 20:35:41.350711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:41.350685    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" event={"ID":"4e8677a8-ebe4-4a37-9a4e-e43551a314ff","Type":"ContainerStarted","Data":"45decad66768620408a903bb86166d25f4ccbbc2152cf146bdcd3009ecd3aeeb"}
Nov 01 20:35:41.385116 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:41.385060    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podStartSLOduration=2.385045308 podStartE2EDuration="2.385045308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:41.383335351 +0000 UTC m=+2186.798967825" watchObservedRunningTime="2026-11-01 20:35:41.385045308 +0000 UTC m=+2186.800677781"
Nov 01 20:35:41.812439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:41.812400    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:42.323727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:42.323686    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:42.325424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:42.325397    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 01 20:35:42.325488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:42.325448    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 01 20:35:42.598229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:42.598128    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:43.323985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:43.323943    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 01 20:35:43.324403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:43.324006    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 01 20:35:43.479325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:43.479288    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:44.294287 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:44.294250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:44.324476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:44.324442    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 01 20:35:44.324915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:44.324493    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 01 20:35:45.083500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:45.083456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:45.324002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:45.323965    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 01 20:35:45.324191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:45.324021    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 01 20:35:45.979003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:45.978964    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:46.324323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:46.324231    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 01 20:35:46.324323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:46.324291    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 01 20:35:46.744282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:46.744247    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:47.324453 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:47.324418    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 01 20:35:47.324865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:47.324476    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 01 20:35:47.358103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:47.358068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:47.966142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:47.966101    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:48.695515 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:48.695481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:48.806751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:48.806706    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:48.923964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:48.923927    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:48.924796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:48.924767    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:48.925516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:48.925494    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh"
Nov 01 20:35:50.696195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:35:50.696160    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:35:59.169180 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:35:59.169141    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:36:11.370143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:11.370093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:11.962394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:11.962343    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:12.578833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:12.578793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:13.144194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:13.144154    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:13.744139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:13.744102    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:14.443806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:14.443772    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:15.050567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:15.050526    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:15.645546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:15.645508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:16.245867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:16.245829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:16.847465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:16.847426    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:17.549771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:17.549730    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:18.160968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:18.160935    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:18.758601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:18.758557    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:19.355659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:19.355608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:20.028342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:20.028304    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:20.596327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:20.596292    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:20.696816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:20.696776    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:21.251605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:21.251572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:21.847455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:21.847419    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:22.462385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:22.462345    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:23.054136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.054103    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:23.286072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.286033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:23.502118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.502082    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"]
Nov 01 20:36:23.502597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.502293    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg" podUID="6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" containerName="manager" containerID="cri-o://70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164" gracePeriod=10
Nov 01 20:36:23.757751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.757686    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:36:23.780190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.780139    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:23.803517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.803481    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume\") pod \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") "
Nov 01 20:36:23.803727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.803673    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm\" (UniqueName: \"kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm\") pod \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\" (UID: \"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66\") "
Nov 01 20:36:23.803985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.803951    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume" pod "6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" (UID: "6f62c32d-0014-4bf3-853c-dd0cfc0f6b66"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:23.806022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.805996    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm" pod "6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" (UID: "6f62c32d-0014-4bf3-853c-dd0cfc0f6b66"). InnerVolumeSpecName "kube-api-access-khhpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:23.906047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.905315    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhpm\" (UniqueName: \"kubernetes.io/projected/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-kube-api-access-khhpm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:36:23.906047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:23.905361    6479 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66-extensions-socket-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:36:24.348974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.348932    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:24.594466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.594429    6479 generic.go:356] "Generic (PLEG): container finished" podID="6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" containerID="70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164" exitCode=0
Nov 01 20:36:24.595022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.594476    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"
Nov 01 20:36:24.595022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.594469    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg" event={"ID":"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66","Type":"ContainerDied","Data":"70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164"}
Nov 01 20:36:24.595022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.594603    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg" event={"ID":"6f62c32d-0014-4bf3-853c-dd0cfc0f6b66","Type":"ContainerDied","Data":"25c30c102954e330c68304fefe64c167f9226716bc334d25e8ff9f31c2b01afa"}
Nov 01 20:36:24.595022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.594626    6479 scope.go:122] "RemoveContainer" containerID="70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164"
Nov 01 20:36:24.610752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.610731    6479 scope.go:122] "RemoveContainer" containerID="70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164"
Nov 01 20:36:24.611018 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:36:24.610998    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164\": container with ID starting with 70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164 not found: ID does not exist" containerID="70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164"
Nov 01 20:36:24.611094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.611026    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164"} err="failed to get container status \"70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164\": rpc error: code = NotFound desc = could not find container \"70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164\": container with ID starting with 70aa21019363e1e753e3a168705eb2630278d40d1637c181420f901e253ee164 not found: ID does not exist"
Nov 01 20:36:24.615966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.615939    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"]
Nov 01 20:36:24.620886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.620864    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59thg"]
Nov 01 20:36:24.706461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:24.706425    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" path="/var/lib/kubelet/pods/6f62c32d-0014-4bf3-853c-dd0cfc0f6b66/volumes"
Nov 01 20:36:25.029864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:25.029797    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:25.558265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:25.558215    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:26.152555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:26.152519    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:26.748725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:26.748692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:27.435496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:27.435458    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:28.047297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:28.047257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:28.647142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:28.647109    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:29.170367 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:36:29.170329    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:36:29.245671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:29.245605    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:29.937430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:29.937393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:30.539905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:30.539866    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:31.153835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:31.153798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:31.749213 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:31.749180    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:32.457267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:32.457230    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:32.951717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:32.951673    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:33.561325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:33.561284    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:34.150223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:34.150189    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:34.768496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:34.768456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:35.400947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:35.400912    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:36.011552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:36.011514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:36.745046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:36.745007    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:37.384864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:37.384829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:37.954911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:37.954872    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:38.546694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:38.546631    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:39.157612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:39.157576    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:39.748914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:39.748877    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:40.350669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:40.350613    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:40.945118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:40.945081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:41.542675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:41.542627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:42.141501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:42.141459    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:42.776840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:42.776799    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:43.343327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:43.343288    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:43.943060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:43.943013    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:44.597437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:44.597389    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:46.535063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:46.535025    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:47.158148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:47.158107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:47.746455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:47.746382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:48.443416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:48.443385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:49.047757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:49.047705    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:49.646139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:49.646044    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:50.251869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:50.251804    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:50.847686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:50.847615    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:51.445295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:51.445260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:52.048188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:52.048157    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:52.644052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:52.644016    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:53.247124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:53.247090    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:53.894318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:53.894283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:54.547932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:54.547891    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:55.147951 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:55.147874    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:55.756775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:55.756735    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:56.453044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:56.453007    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:57.054784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:57.054752    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:57.659733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:57.659696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:58.286500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:58.286443    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:58.849252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:58.849220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:36:59.170391 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:36:59.170303    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:36:59.438208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:36:59.438119    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:00.064557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:00.064513    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:00.747169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:00.747134    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:01.350992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:01.350961    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:01.954175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:01.954138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:02.555614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:02.555574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:03.224955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:03.224921    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:03.927894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:03.927858    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:04.445366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:04.445328    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:05.044001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:05.043963    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:05.650142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:05.650107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:06.262120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:06.262082    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:06.862588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:06.862556    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:07.449094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:07.449054    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:08.047401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:08.047367    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:08.299667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:08.299558    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:08.399795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:08.399757    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:08.712974 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:08.712938    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:09.359586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:09.359551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:10.042784 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:10.042751    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:10.641445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:10.641407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:11.244914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:11.244874    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:11.847178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:11.847145    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:12.452804 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:12.452767    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:13.049234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:13.049200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:13.540131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:13.540092    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:13.641079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:13.641047    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:13.741516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:13.741480    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:14.407192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:14.407144    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:14.949669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:14.949611    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:15.547458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:15.547406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:16.149388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:16.149350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:16.749083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:16.749038    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:17.371792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:17.371754    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:17.953727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:17.953685    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:18.569297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:18.569259    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:19.194790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:19.194762    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:19.747149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:19.747113    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:20.357246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:20.357202    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:21.018904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:21.018825    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:21.660433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:21.660398    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:22.296626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:22.296591    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:22.859184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:22.859146    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:23.272687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:23.272632    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:23.457841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:23.457810    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:24.133690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.133637    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:24.563503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.563465    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"]
Nov 01 20:37:24.564093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.564071    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" containerName="manager"
Nov 01 20:37:24.564193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.564095    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" containerName="manager"
Nov 01 20:37:24.564254 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.564193    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f62c32d-0014-4bf3-853c-dd0cfc0f6b66" containerName="manager"
Nov 01 20:37:24.568575 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.568552    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.573367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.573348    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-swjbx\""
Nov 01 20:37:24.578625 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.578603    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"]
Nov 01 20:37:24.636693 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.636641    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5fs\" (UniqueName: \"kubernetes.io/projected/c59e357a-a3be-4269-9cc3-de751b07d38a-kube-api-access-zx5fs\") pod \"kuadrant-operator-controller-manager-678d87766f-dvwgc\" (UID: \"c59e357a-a3be-4269-9cc3-de751b07d38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.636825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.636717    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c59e357a-a3be-4269-9cc3-de751b07d38a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dvwgc\" (UID: \"c59e357a-a3be-4269-9cc3-de751b07d38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.737752 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.737716    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5fs\" (UniqueName: \"kubernetes.io/projected/c59e357a-a3be-4269-9cc3-de751b07d38a-kube-api-access-zx5fs\") pod \"kuadrant-operator-controller-manager-678d87766f-dvwgc\" (UID: \"c59e357a-a3be-4269-9cc3-de751b07d38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.737903 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.737759    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c59e357a-a3be-4269-9cc3-de751b07d38a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dvwgc\" (UID: \"c59e357a-a3be-4269-9cc3-de751b07d38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.738162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.738141    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c59e357a-a3be-4269-9cc3-de751b07d38a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dvwgc\" (UID: \"c59e357a-a3be-4269-9cc3-de751b07d38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.745729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.745700    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5fs\" (UniqueName: \"kubernetes.io/projected/c59e357a-a3be-4269-9cc3-de751b07d38a-kube-api-access-zx5fs\") pod \"kuadrant-operator-controller-manager-678d87766f-dvwgc\" (UID: \"c59e357a-a3be-4269-9cc3-de751b07d38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.883170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.883079    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:24.963730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:24.963691    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:25.164790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:25.164753    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"]
Nov 01 20:37:25.167863 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:37:25.167833    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59e357a_a3be_4269_9cc3_de751b07d38a.slice/crio-a55196fedb1aa4dca1dae2727360d6d5385e39a1a888a8f1c08d259f4eb98917 WatchSource:0}: Error finding container a55196fedb1aa4dca1dae2727360d6d5385e39a1a888a8f1c08d259f4eb98917: Status 404 returned error can't find the container with id a55196fedb1aa4dca1dae2727360d6d5385e39a1a888a8f1c08d259f4eb98917
Nov 01 20:37:25.839855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:25.839818    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:25.909123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:25.909084    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc" event={"ID":"c59e357a-a3be-4269-9cc3-de751b07d38a","Type":"ContainerStarted","Data":"0e4594f4d0ddcc0d6a93fb560b448c0f10ec4e8d03d1be7e3c6c0f01db37655c"}
Nov 01 20:37:25.909337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:25.909151    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc" event={"ID":"c59e357a-a3be-4269-9cc3-de751b07d38a","Type":"ContainerStarted","Data":"a55196fedb1aa4dca1dae2727360d6d5385e39a1a888a8f1c08d259f4eb98917"}
Nov 01 20:37:25.909722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:25.909697    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:25.930227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:25.930169    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc" podStartSLOduration=1.9301539540000001 podStartE2EDuration="1.930153954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:25.928209582 +0000 UTC m=+2291.343842047" watchObservedRunningTime="2026-11-01 20:37:25.930153954 +0000 UTC m=+2291.345786427"
Nov 01 20:37:26.566612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:26.566575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:27.449018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:27.448984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:28.150081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:28.150051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:28.750744 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:28.750713    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:29.169605 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:37:29.169520    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:37:29.350228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:29.350196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:30.003910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:30.003875    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:30.665257 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:30.665218    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:31.265550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:31.265510    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:31.852081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:31.852042    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:32.585352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:32.585316    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:33.149407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:33.149370    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:33.761402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:33.761368    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:34.351449 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:34.351418    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:35.011275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:35.011236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:35.646828 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:35.646792    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:36.249816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:36.249780    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:36.854007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:36.853974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:36.922360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:36.922326    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dvwgc"
Nov 01 20:37:37.663846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:37.663811    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:38.459879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:38.459843    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:39.056659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:39.056617    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:39.660954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:39.660915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:40.254962 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:40.254927    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:40.859328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:40.859296    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:41.456910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:41.456874    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:42.049612 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:42.049575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:42.655567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:42.655530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:43.252106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:43.252069    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:43.853330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:43.853291    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:44.452563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:44.452518    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:45.054820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:45.054776    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:45.737550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:45.737510    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:45.910593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:45.910555    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:37:46.425930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:46.425882    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:47.164979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:47.164939    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:47.941129 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:47.941091    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:48.877709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:48.877674    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:49.535558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:49.535521    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:50.239312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:50.239278    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:50.862056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:50.861989    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:51.531432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:51.531397    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:52.141430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:52.141393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:52.749810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:52.749764    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:53.440609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:53.440573    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:54.048549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:54.048508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:54.758946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:54.758906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:55.353009 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:55.352971    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:55.962768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:55.962735    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:56.559412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:56.559348    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:57.161826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:57.161788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:57.758954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:57.758904    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:58.440264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:58.440192    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:59.041111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:59.041022    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:37:59.173501 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:37:59.173466    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:37:59.644627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:37:59.644588    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:00.372137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:00.372109    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:01.125774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:01.125740    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:38:01.153602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:01.153570    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:01.762128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:01.762035    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:02.362742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:02.362690    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:03.057234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:03.057194    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:03.751499 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:03.751460    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:04.473569 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:04.473536    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:05.223678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:05.220104    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:06.084168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:06.084091    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:06.305325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:06.305287    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:38:06.833670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:06.832692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:07.449411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:07.449377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:08.053829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:08.053786    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:08.831253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:08.831220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:09.536999 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:09.536964    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:10.415922 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:10.415872    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:11.318021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:11.317984    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:12.069657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:12.069602    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:12.662248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:12.662188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:13.258780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:13.258692    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:13.941459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:13.941401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:14.045669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:14.045104    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:14.149278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:14.149203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:14.857573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:14.857538    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:15.631584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:15.631543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:16.344334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:16.344305    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:16.967875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:16.967833    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:17.347534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:17.347445    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:38:17.543520 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:17.543486    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:18.239787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:18.239742    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:19.076290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:19.076229    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:19.969672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:19.962128    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:20.019837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:20.019786    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:20.020168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:20.020135    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:20.799923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:20.799360    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:20.799923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:20.799466    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:20.969739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:20.969542    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:21.567411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:21.567346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:22.161038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:22.160996    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:22.863629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:22.863578    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:23.317315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:23.317244    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:23.565137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:23.565097    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:24.336327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:24.336293    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:25.061686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:25.061540    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:25.658191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:25.658149    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:26.256615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:26.256585    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:26.960220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:26.960190    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:27.544777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:27.544740    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:28.153289 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:28.153258    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:28.752156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:28.752123    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:29.169351 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:38:29.169271    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:38:29.358089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:29.358051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:30.063463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:30.063428    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:30.667771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:30.667737    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:31.262073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:31.262026    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:31.862101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:31.862048    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:32.470525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:32.470490    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:33.314687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:33.314632    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:33.944108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:33.944077    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:34.556366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:34.556321    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:34.641329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:34.641294    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:38:35.334902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:35.334857    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:36.042698 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:36.036479    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:37.002248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:37.002196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:37.782131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:37.782093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:38.446750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:38.446719    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:39.052067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:39.052022    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:39.644438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:39.644404    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:40.242185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:40.242151    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:40.853991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:40.853957    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:41.463438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:41.463400    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:42.058259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:42.058214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:42.723980 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:42.723939    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:43.319223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:43.319185    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:43.957343 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:43.957311    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:44.567932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:44.567896    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:45.251065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:45.251032    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:45.852297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:45.852262    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:46.453724 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:46.453686    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:47.133562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:47.133524    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:47.746072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:47.746039    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:48.361003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:48.360966    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:48.950682 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:48.950626    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:49.539161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:49.539119    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:50.247104 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:50.247063    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:50.840796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:50.840761    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:51.533523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:51.533477    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:52.055285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:52.055250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:52.658012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:52.657978    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:53.252709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:53.252674    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:53.847633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:53.847595    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:54.449735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:54.449697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:55.060915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:55.060860    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:55.160432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:55.160393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:55.656958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:55.656920    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:56.470539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:56.470502    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:57.457331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:57.457283    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:58.059146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:58.059111    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:58.665396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:58.665362    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:38:59.169011 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:38:59.168978    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:38:59.256111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:38:59.256080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:00.056201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:00.056165    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:00.640666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:00.640622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:01.255322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:01.255288    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:01.851594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:01.851564    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:02.457736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:02.457697    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:03.062305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:03.062266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:03.828495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:03.828464    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:04.455516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:04.455484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:05.055895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:05.055853    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:05.664530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:05.664471    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:06.467576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:06.467537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:07.274670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:07.267731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:08.039721 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:08.039682    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:08.652984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:08.652954    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:09.336522 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:09.336484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:09.844615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:09.844568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:09.943965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:09.943925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:10.044440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:10.044407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:10.549262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:10.549214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:10.649443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:10.649410    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:11.242010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:11.241974    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:11.844029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:11.843988    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:12.444843 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:12.444806    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:13.048473 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:13.048441    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:13.658082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:13.658049    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:14.330084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:14.330047    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:14.945433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:14.945396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:15.629052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:15.629008    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:16.316188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:16.316126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:17.151679 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:17.151126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:17.996669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:17.995134    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:18.171905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:18.171857    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:39:18.172275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:18.172237    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:39:18.172909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:18.172870    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:39:18.173088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:18.173067    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:39:18.741773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:18.741733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:19.346512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:19.346478    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:19.952757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:19.952724    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:20.551671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:20.551611    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:21.149365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:21.149330    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:21.750736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:21.750695    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:22.349081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:22.349051    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:22.951091 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:22.951045    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:23.365527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:23.365443    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:23.544327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:23.544297    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:24.317880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:24.317846    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:24.959452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:24.959413    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:25.555597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:25.555550    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:26.234753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:26.234717    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:27.082434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:27.082377    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:28.077161 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:28.077122    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:28.737994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:28.737959    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:29.169122 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:39:29.169041    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:39:29.353963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:29.353931    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:29.956968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:29.956938    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:30.629610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:30.629579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:31.241424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:31.241391    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:31.849132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:31.849099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:32.547800 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:32.547770    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:33.146738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:33.146704    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:33.760421 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:33.760386    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:34.358304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:34.358272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:35.058333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:35.058298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:35.723760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:35.723723    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:36.452939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:36.452897    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:37.357701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:37.357662    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:38.243169 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:38.243130    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:38.938435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:38.938401    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:39.546832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:39.546797    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:40.145741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:40.145709    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:40.760756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:40.760724    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:41.351337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:41.351302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:41.948401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:41.948368    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:42.544066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:42.544017    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:43.145884 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:43.145837    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:43.853355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:43.853325    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:44.345675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:44.345629    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:44.964985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:44.964950    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:45.637275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:45.637236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:46.261521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:46.261479    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:47.072348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:47.072295    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:47.985875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:47.985835    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:48.198879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:48.198825    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:39:48.199547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:48.199513    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:39:48.746214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:48.746175    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:49.341826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:49.341786    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:49.953059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:49.953025    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:50.550064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:50.550032    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:51.157876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:51.157843    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:51.763223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:51.763185    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:52.355328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:52.355298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:52.953941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:52.953889    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:53.553210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:53.553172    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:54.155334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:54.155297    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:54.754324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:54.754280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:55.461887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:55.461841    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:56.242971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:56.242932    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:56.857382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:56.857344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:57.653270 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:57.653228    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:58.491584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:58.491542    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:59.159246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:59.159171    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:39:59.190918 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:39:59.190836    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:39:59.749993 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:39:59.749959    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:00.348482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:00.348447    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:00.964674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:00.964598    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:01.563839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:01.563801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:02.153783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:02.153748    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:02.750235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:02.750200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:03.346846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:03.346810    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:03.947292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:03.947255    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:04.647660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:04.647611    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:05.360073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:05.360033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:05.958873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:05.958836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:06.575534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:06.575498    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:07.237699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:07.237621    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:08.090044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:08.090010    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:09.140814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:09.140784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:09.747948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:09.747910    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:10.350060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:10.350027    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:10.952873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:10.952838    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:11.555838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:11.555802    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:12.252603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:12.252569    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:12.849607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:12.849574    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:13.450294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:13.450264    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:14.054413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:14.054381    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:14.759671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:14.759619    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:15.366050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:15.366010    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:16.123923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:16.123886    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:16.753746 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:16.753708    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:17.572233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:17.572191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:18.161666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:18.160881    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:40:18.161666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:18.160957    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:40:18.161666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:18.161455    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:40:18.161666 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:18.161564    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:40:18.491536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:18.491434    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:19.156390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:19.156346    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:19.755672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:19.755624    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:20.358428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:20.358381    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:20.958751 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:20.958718    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:21.558704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:21.558667    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:22.157820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:22.157787    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:22.752234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:22.752201    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:23.355609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:23.355573    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:23.459446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:23.459393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:23.961517 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:23.961474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:24.571050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:24.571015    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:25.151108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:25.151072    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:25.844512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:25.844476    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:26.458082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:26.458052    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:40:26.586412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:26.586382    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:27.338968 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:27.338928    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:27.652244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:27.652110    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:28.356531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:28.356496    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:29.051673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:29.051595    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:29.209485 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:40:29.209408    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:40:29.745307 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:29.745272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:30.445265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:30.445231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:31.050525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:31.050491    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:31.654268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:31.654235    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:32.263529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:32.263494    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:32.718366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:32.718323    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:32.933370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:32.933337    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:33.560889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:33.560855    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:34.144337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:34.144298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:34.756802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:34.756767    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:35.347926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:35.347893    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:36.051824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:36.051788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:36.753867 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:36.753836    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:37.437155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:37.437115    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:37.952952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:37.952919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:38.557567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:38.557533    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:39.160346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:39.160309    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:39.774273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:39.774239    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:40.361662 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:40.361595    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:40.954171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:40.954132    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:41.141700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:41.141662    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:41.241761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:41.241668    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:41.562164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:41.562084    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:42.152485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:42.152449    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:42.750956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:42.750921    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:43.068808 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:43.068715    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:43.432586 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:43.432506    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:43.834172 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:43.834135    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:40:44.144932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:44.144824    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:44.826678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:44.826631    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:45.460150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:45.460114    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:46.162610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:46.162575    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:46.640757 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:46.640720    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:47.246350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:47.246318    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:47.848183 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:47.848143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:48.461875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:48.461831    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:49.178611 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:49.178574    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"]
Nov 01 20:40:49.210681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:49.206570    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"]
Nov 01 20:40:49.243441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:49.243391    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:49.343985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:49.343951    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:49.453470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:49.453434    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:49.947329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:49.947299    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:54.179255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.179209    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl" podUID="3ae245b6-740c-48da-a939-291af7712863" containerName="payload-processing" containerID="cri-o://2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8" gracePeriod=25
Nov 01 20:40:54.207324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.207290    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj" podUID="12b91524-aa26-4e14-9e16-a390623fd08c" containerName="payload-pre-processing" containerID="cri-o://7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b" gracePeriod=25
Nov 01 20:40:54.445706 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.445684    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:40:54.481772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.481750    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:40:54.549985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.549950    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume\") pod \"12b91524-aa26-4e14-9e16-a390623fd08c\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") "
Nov 01 20:40:54.550174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550096    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume\") pod \"3ae245b6-740c-48da-a939-291af7712863\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") "
Nov 01 20:40:54.550256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550203    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh\" (UniqueName: \"kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh\") pod \"3ae245b6-740c-48da-a939-291af7712863\" (UID: \"3ae245b6-740c-48da-a939-291af7712863\") "
Nov 01 20:40:54.550320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550272    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw\" (UniqueName: \"kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw\") pod \"12b91524-aa26-4e14-9e16-a390623fd08c\" (UID: \"12b91524-aa26-4e14-9e16-a390623fd08c\") "
Nov 01 20:40:54.550494 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550460    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume" pod "12b91524-aa26-4e14-9e16-a390623fd08c" (UID: "12b91524-aa26-4e14-9e16-a390623fd08c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:54.550590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550515    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume" pod "3ae245b6-740c-48da-a939-291af7712863" (UID: "3ae245b6-740c-48da-a939-291af7712863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:54.550699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550672    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b91524-aa26-4e14-9e16-a390623fd08c-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:54.550851 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.550703    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae245b6-740c-48da-a939-291af7712863-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:54.552601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.552573    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw" pod "12b91524-aa26-4e14-9e16-a390623fd08c" (UID: "12b91524-aa26-4e14-9e16-a390623fd08c"). InnerVolumeSpecName "kube-api-access-klfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:54.552739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.552716    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh" pod "3ae245b6-740c-48da-a939-291af7712863" (UID: "3ae245b6-740c-48da-a939-291af7712863"). InnerVolumeSpecName "kube-api-access-2q9fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:54.638931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.638902    6479 generic.go:356] "Generic (PLEG): container finished" podID="12b91524-aa26-4e14-9e16-a390623fd08c" containerID="7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b" exitCode=0
Nov 01 20:40:54.639100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.638952    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"
Nov 01 20:40:54.639100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.638982    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj" event={"ID":"12b91524-aa26-4e14-9e16-a390623fd08c","Type":"ContainerDied","Data":"7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b"}
Nov 01 20:40:54.639100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.639076    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj" event={"ID":"12b91524-aa26-4e14-9e16-a390623fd08c","Type":"ContainerDied","Data":"424db1481fac4a3699db089e1f363d89d29851e181437534628342cbbef27f99"}
Nov 01 20:40:54.639275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.639111    6479 scope.go:122] "RemoveContainer" containerID="7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b"
Nov 01 20:40:54.640467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.640448    6479 generic.go:356] "Generic (PLEG): container finished" podID="3ae245b6-740c-48da-a939-291af7712863" containerID="2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8" exitCode=0
Nov 01 20:40:54.640564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.640502    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl" event={"ID":"3ae245b6-740c-48da-a939-291af7712863","Type":"ContainerDied","Data":"2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8"}
Nov 01 20:40:54.640564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.640514    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"
Nov 01 20:40:54.640564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.640531    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl" event={"ID":"3ae245b6-740c-48da-a939-291af7712863","Type":"ContainerDied","Data":"9bc78d0a4667c7209091727d158d51f3a56b140c5d99d33a96bbe4e7daa46a19"}
Nov 01 20:40:54.650552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.650534    6479 scope.go:122] "RemoveContainer" containerID="7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b"
Nov 01 20:40:54.650873 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:40:54.650844    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b\": container with ID starting with 7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b not found: ID does not exist" containerID="7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b"
Nov 01 20:40:54.650981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.650878    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b"} err="failed to get container status \"7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b\": rpc error: code = NotFound desc = could not find container \"7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b\": container with ID starting with 7c5f626c301b6e5c199fa64de370833c6dc434023c0adbdd51c843a76b3d6d2b not found: ID does not exist"
Nov 01 20:40:54.650981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.650895    6479 scope.go:122] "RemoveContainer" containerID="2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8"
Nov 01 20:40:54.651251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.651232    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9fh\" (UniqueName: \"kubernetes.io/projected/3ae245b6-740c-48da-a939-291af7712863-kube-api-access-2q9fh\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:54.651322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.651260    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfdw\" (UniqueName: \"kubernetes.io/projected/12b91524-aa26-4e14-9e16-a390623fd08c-kube-api-access-klfdw\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:54.659490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.659018    6479 scope.go:122] "RemoveContainer" containerID="2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8"
Nov 01 20:40:54.659490 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:40:54.659341    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8\": container with ID starting with 2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8 not found: ID does not exist" containerID="2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8"
Nov 01 20:40:54.659490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.659376    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8"} err="failed to get container status \"2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8\": rpc error: code = NotFound desc = could not find container \"2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8\": container with ID starting with 2e7d0d2b50aa36399bc387548b1a5d35090df83fffdff184308cf303ce068cd8 not found: ID does not exist"
Nov 01 20:40:54.660730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.660706    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"]
Nov 01 20:40:54.665916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.665895    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-1bfa0e-778b5ffcff-xrldj"]
Nov 01 20:40:54.674311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.674289    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"]
Nov 01 20:40:54.678228 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.678208    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-1bfa0e-54bdfb5cb5-bbjkl"]
Nov 01 20:40:54.707066 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.707042    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b91524-aa26-4e14-9e16-a390623fd08c" path="/var/lib/kubelet/pods/12b91524-aa26-4e14-9e16-a390623fd08c/volumes"
Nov 01 20:40:54.707411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:54.707398    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae245b6-740c-48da-a939-291af7712863" path="/var/lib/kubelet/pods/3ae245b6-740c-48da-a939-291af7712863/volumes"
Nov 01 20:40:55.095985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:55.095862    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:55.481864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:55.481790    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:55.557321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:40:55.557282    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:40:59.168354 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:40:59.168316    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:41:00.262268 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:00.262235    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:41:00.590122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:00.589640    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"]
Nov 01 20:41:00.590122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:00.589950    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" podUID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" containerName="maas-api" containerID="cri-o://8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b" gracePeriod=30
Nov 01 20:41:00.628300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:00.628256    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:00.840874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:00.840784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:00.906979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:00.906952    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:41:01.035117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.033470    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\") pod \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") "
Nov 01 20:41:01.035117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.033521    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs\") pod \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") "
Nov 01 20:41:01.035117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.033617    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca\") pod \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") "
Nov 01 20:41:01.035117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.033700    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv\" (UniqueName: \"kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv\") pod \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\" (UID: \"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e\") "
Nov 01 20:41:01.035774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.035745    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca" pod "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" (UID: "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:01.036802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.036778    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs" pod "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" (UID: "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:01.036906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.036873    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv" pod "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" (UID: "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e"). InnerVolumeSpecName "kube-api-access-bn2xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:01.037452 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.037430    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls" pod "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" (UID: "6f4e8ca3-61ec-43e5-87a7-b38f21316b1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:01.135539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.135446    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:01.135539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.135490    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:01.135539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.135505    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:01.135539 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.135519    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2xv\" (UniqueName: \"kubernetes.io/projected/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e-kube-api-access-bn2xv\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:01.402178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.402098    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:01.677893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.677800    6479 generic.go:356] "Generic (PLEG): container finished" podID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" containerID="8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b" exitCode=0
Nov 01 20:41:01.677893 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.677844    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"
Nov 01 20:41:01.678131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.677849    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" event={"ID":"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e","Type":"ContainerDied","Data":"8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b"}
Nov 01 20:41:01.678131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.677982    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" event={"ID":"6f4e8ca3-61ec-43e5-87a7-b38f21316b1e","Type":"ContainerDied","Data":"d6b46acc8c9891d7488982465f08d1682460f735cc9ae0a5a7bd3d6c84f32bee"}
Nov 01 20:41:01.678131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.678018    6479 scope.go:122] "RemoveContainer" containerID="8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b"
Nov 01 20:41:01.688150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.688131    6479 scope.go:122] "RemoveContainer" containerID="8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b"
Nov 01 20:41:01.688409 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:01.688392    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b\": container with ID starting with 8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b not found: ID does not exist" containerID="8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b"
Nov 01 20:41:01.688471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.688416    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b"} err="failed to get container status \"8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b\": rpc error: code = NotFound desc = could not find container \"8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b\": container with ID starting with 8e2fd28d439603861c90c626412f5cc4d1f63e4be86cfe9736e2b3187bb0365b not found: ID does not exist"
Nov 01 20:41:01.698606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.698577    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"]
Nov 01 20:41:01.702414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:01.702384    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b"]
Nov 01 20:41:02.706286 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:02.706252    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" path="/var/lib/kubelet/pods/6f4e8ca3-61ec-43e5-87a7-b38f21316b1e/volumes"
Nov 01 20:41:03.898352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:03.898302    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-1bfa0e-865fdbf75-6gg7b" podUID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.187:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:41:05.139160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:05.139124    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:41:06.277758 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:06.277728    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:06.351564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:06.351513    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:09.554330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:09.554289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:09.682237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:09.682193    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:10.234583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:10.234547    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:11.103050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:11.103014    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:11.420040 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:11.419954    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"]
Nov 01 20:41:11.420220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:11.420174    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy" containerID="cri-o://b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727" gracePeriod=30
Nov 01 20:41:11.455694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:11.455642    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:15.655956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:15.655919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:15.757411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:15.757378    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:15.861016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:15.860964    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:16.018490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.018447    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"]
Nov 01 20:41:16.019011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.018979    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" containerName="maas-api"
Nov 01 20:41:16.019011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019000    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" containerName="maas-api"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019034    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3ae245b6-740c-48da-a939-291af7712863" containerName="payload-processing"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019041    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae245b6-740c-48da-a939-291af7712863" containerName="payload-processing"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019056    6479 cpu_manager.go:414] "containerMap: removing container" podUID="12b91524-aa26-4e14-9e16-a390623fd08c" containerName="payload-pre-processing"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019065    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b91524-aa26-4e14-9e16-a390623fd08c" containerName="payload-pre-processing"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019148    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae245b6-740c-48da-a939-291af7712863" containerName="payload-processing"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019156    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b91524-aa26-4e14-9e16-a390623fd08c" containerName="payload-pre-processing"
Nov 01 20:41:16.019202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.019167    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4e8ca3-61ec-43e5-87a7-b38f21316b1e" containerName="maas-api"
Nov 01 20:41:16.022280 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.022260    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.026470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.026436    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-022966d4-openshift-default-dockercfg-nbl6m\""
Nov 01 20:41:16.034836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.034810    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"]
Nov 01 20:41:16.061456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.061407    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:16.091979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.091947    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.091987    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092099 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092014    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092123    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092185    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092238    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092298    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8xgj\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092370    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.092459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.092440    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.161152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.161117    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:16.194819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.194778    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8xgj\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.194850    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.194955    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.194982    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.195017    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.195070    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.195099    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.195153    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.195180    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.195732 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.195706    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.196513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.196488    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.196976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.196952    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.198151 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.198126    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.199044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.199021    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.237684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.231838    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.237684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.233125    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.237684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.236573    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.237684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.237037    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8xgj\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj\") pod \"e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") " pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.341500 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.341408    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:16.630921 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:41:16.630889    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02460fc7_dd51_42a7_8e6d_dde799a5c428.slice/crio-30e263dd961e7b5d9a1386d29dd206d4a567b9930db3a677f78ea807b01d530d WatchSource:0}: Error finding container 30e263dd961e7b5d9a1386d29dd206d4a567b9930db3a677f78ea807b01d530d: Status 404 returned error can't find the container with id 30e263dd961e7b5d9a1386d29dd206d4a567b9930db3a677f78ea807b01d530d
Nov 01 20:41:16.631204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.631181    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"]
Nov 01 20:41:16.633399 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.633367    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:41:16.633507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.633441    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:41:16.633507 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.633470    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:41:16.682938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.682912    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:41:16.702290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702005    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.702290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702050    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.702290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702099    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.702290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702168    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.702290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702216    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.703467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702768    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.703467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702848    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.703467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702893    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.703467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.702966    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data\") pod \"71a249be-eaaf-43d0-a740-e92466687b8e\" (UID: \"71a249be-eaaf-43d0-a740-e92466687b8e\") "
Nov 01 20:41:16.703937 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.703906    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.704026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.703916    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.706119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.706093    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.706238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.706113    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:16.707126 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.707101    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.708919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.708890    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:16.713222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.713173    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.714849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.714815    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "kube-api-access-f4mmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:16.717135 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.717108    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo" pod "71a249be-eaaf-43d0-a740-e92466687b8e" (UID: "71a249be-eaaf-43d0-a740-e92466687b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:41:16.760512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.760468    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" event={"ID":"02460fc7-dd51-42a7-8e6d-dde799a5c428","Type":"ContainerStarted","Data":"cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"}
Nov 01 20:41:16.760748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.760585    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" event={"ID":"02460fc7-dd51-42a7-8e6d-dde799a5c428","Type":"ContainerStarted","Data":"30e263dd961e7b5d9a1386d29dd206d4a567b9930db3a677f78ea807b01d530d"}
Nov 01 20:41:16.762776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.762741    6479 generic.go:356] "Generic (PLEG): container finished" podID="71a249be-eaaf-43d0-a740-e92466687b8e" containerID="b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727" exitCode=0
Nov 01 20:41:16.762900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.762839    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" event={"ID":"71a249be-eaaf-43d0-a740-e92466687b8e","Type":"ContainerDied","Data":"b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727"}
Nov 01 20:41:16.762900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.762888    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld" event={"ID":"71a249be-eaaf-43d0-a740-e92466687b8e","Type":"ContainerDied","Data":"1c6d41cefde9b5a83907bbffa62f88cb7c6f29c744ad82c3d7ad9c18113fb2ea"}
Nov 01 20:41:16.763062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.762928    6479 scope.go:122] "RemoveContainer" containerID="b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727"
Nov 01 20:41:16.764727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.763130    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"
Nov 01 20:41:16.776193 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.776167    6479 scope.go:122] "RemoveContainer" containerID="f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"
Nov 01 20:41:16.780847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.780794    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podStartSLOduration=0.7807793 podStartE2EDuration="780.7793ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:16.778532678 +0000 UTC m=+2522.194165152" watchObservedRunningTime="2026-11-01 20:41:16.7807793 +0000 UTC m=+2522.196411776"
Nov 01 20:41:16.788232 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.788211    6479 scope.go:122] "RemoveContainer" containerID="b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727"
Nov 01 20:41:16.788540 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:16.788517    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727\": container with ID starting with b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727 not found: ID does not exist" containerID="b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727"
Nov 01 20:41:16.788632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.788551    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727"} err="failed to get container status \"b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727\": rpc error: code = NotFound desc = could not find container \"b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727\": container with ID starting with b5b406501be63c27596bb60e82f2b2e24b88669402ef86e48d3311686e581727 not found: ID does not exist"
Nov 01 20:41:16.788632 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.788573    6479 scope.go:122] "RemoveContainer" containerID="f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"
Nov 01 20:41:16.788856 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:16.788833    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789\": container with ID starting with f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789 not found: ID does not exist" containerID="f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"
Nov 01 20:41:16.788952 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.788859    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789"} err="failed to get container status \"f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789\": rpc error: code = NotFound desc = could not find container \"f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789\": container with ID starting with f02cc0cc65c005dd14ab61336310b48e80e166a2ae7c0c5d349ab3495806c789 not found: ID does not exist"
Nov 01 20:41:16.792117 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.792094    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"]
Nov 01 20:41:16.793917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.793896    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-zstld"]
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804005    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804036    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mmn\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-kube-api-access-f4mmn\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804053    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a249be-eaaf-43d0-a740-e92466687b8e-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804068    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804082    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804097    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804112    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a249be-eaaf-43d0-a740-e92466687b8e-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804128    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a249be-eaaf-43d0-a740-e92466687b8e-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.804167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:16.804143    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a249be-eaaf-43d0-a740-e92466687b8e-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:17.341988 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:17.341957    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:17.347573 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:17.347542    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:17.770230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:17.770190    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:17.771267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:17.771243    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:18.187123 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:18.187027    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:18.287131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:18.287091    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:18.706593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:18.706560    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" path="/var/lib/kubelet/pods/71a249be-eaaf-43d0-a740-e92466687b8e/volumes"
Nov 01 20:41:19.343181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343149    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"]
Nov 01 20:41:19.343681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343633    6479 cpu_manager.go:414] "containerMap: removing container" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy"
Nov 01 20:41:19.343738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343684    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy"
Nov 01 20:41:19.343738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343700    6479 cpu_manager.go:414] "containerMap: removing container" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy"
Nov 01 20:41:19.343738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343705    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy"
Nov 01 20:41:19.343861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343799    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy"
Nov 01 20:41:19.343861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.343809    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a249be-eaaf-43d0-a740-e92466687b8e" containerName="istio-proxy"
Nov 01 20:41:19.348432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.348409    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.359434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.357822    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-022966d4-serving-cert\""
Nov 01 20:41:19.359434 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.358408    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"]
Nov 01 20:41:19.383629 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.383592    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:19.434620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.434590    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"]
Nov 01 20:41:19.437377 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.437350    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.437533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.437414    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.437533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.437506    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.437636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.437547    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj7v\" (UniqueName: \"kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.438072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.438059    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.447029 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.446977    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-022966d4\""
Nov 01 20:41:19.448929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.448908    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"]
Nov 01 20:41:19.457601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.457576    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-022966d4-dockercfg-m7qlp\""
Nov 01 20:41:19.484083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.484055    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"]
Nov 01 20:41:19.484487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.484457    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:19.487527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.487510    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.497477 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.497454    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"]
Nov 01 20:41:19.566247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.566211    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqnv6\" (UniqueName: \"kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6\") pod \"payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.566541 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.566516    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.567497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.567473    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2s4p\" (UniqueName: \"kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p\") pod \"payload-processing-e2e-ait-022966d4-977867645-x62z2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.567675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.567639    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.567831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.567813    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.567994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.567979    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.568110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.568090    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-022966d4-977867645-x62z2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.568191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.568135    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj7v\" (UniqueName: \"kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.568534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.567406    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.577088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.577059    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj7v\" (UniqueName: \"kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.577561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.577536    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.591484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.591455    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls\") pod \"maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.668264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.668174    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:19.669562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.669528    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqnv6\" (UniqueName: \"kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6\") pod \"payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.669741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.669600    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2s4p\" (UniqueName: \"kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p\") pod \"payload-processing-e2e-ait-022966d4-977867645-x62z2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.669741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.669633    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.669741 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.669706    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-022966d4-977867645-x62z2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.670391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.670366    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.670483 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.670373    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume\") pod \"payload-processing-e2e-ait-022966d4-977867645-x62z2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.677674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.677632    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2s4p\" (UniqueName: \"kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p\") pod \"payload-processing-e2e-ait-022966d4-977867645-x62z2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") " pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.677771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.677725    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqnv6\" (UniqueName: \"kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6\") pod \"payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.752866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.752833    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:19.802868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.802827    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:19.942185 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.941639    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"]
Nov 01 20:41:19.944762 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:41:19.944722    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacce6506_4fa8_4c44_afee_a94a9e6b7161.slice/crio-2d11eb044c283590a9d570bc89bc65c6c0ec06a8b3a936e3ec176ac4733addd2 WatchSource:0}: Error finding container 2d11eb044c283590a9d570bc89bc65c6c0ec06a8b3a936e3ec176ac4733addd2: Status 404 returned error can't find the container with id 2d11eb044c283590a9d570bc89bc65c6c0ec06a8b3a936e3ec176ac4733addd2
Nov 01 20:41:19.946252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:19.946233    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:41:20.046592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.046490    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"]
Nov 01 20:41:20.048817 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:41:20.048779    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf97345_657f_4940_84aa_66b87d8e931f.slice/crio-dfe75e3e44cdedbd3d516b8db7fb31932cac7e66d9abfed7a0624cf6042a1a8d WatchSource:0}: Error finding container dfe75e3e44cdedbd3d516b8db7fb31932cac7e66d9abfed7a0624cf6042a1a8d: Status 404 returned error can't find the container with id dfe75e3e44cdedbd3d516b8db7fb31932cac7e66d9abfed7a0624cf6042a1a8d
Nov 01 20:41:20.099128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.099097    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"]
Nov 01 20:41:20.103734 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:41:20.103626    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a573777_e7f2_4616_8857_06c70e61e5c2.slice/crio-5d705fe018fe183dd05c318daca7b5edcb8de787adbc72cb89deecf1cb779998 WatchSource:0}: Error finding container 5d705fe018fe183dd05c318daca7b5edcb8de787adbc72cb89deecf1cb779998: Status 404 returned error can't find the container with id 5d705fe018fe183dd05c318daca7b5edcb8de787adbc72cb89deecf1cb779998
Nov 01 20:41:20.130316 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.130282    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:20.787285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.787180    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx" event={"ID":"acce6506-4fa8-4c44-afee-a94a9e6b7161","Type":"ContainerStarted","Data":"e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f"}
Nov 01 20:41:20.787285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.787255    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:20.787285 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.787284    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx" event={"ID":"acce6506-4fa8-4c44-afee-a94a9e6b7161","Type":"ContainerStarted","Data":"2d11eb044c283590a9d570bc89bc65c6c0ec06a8b3a936e3ec176ac4733addd2"}
Nov 01 20:41:20.788922 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.788895    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg" event={"ID":"3bf97345-657f-4940-84aa-66b87d8e931f","Type":"ContainerStarted","Data":"3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95"}
Nov 01 20:41:20.789052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.788944    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg" event={"ID":"3bf97345-657f-4940-84aa-66b87d8e931f","Type":"ContainerStarted","Data":"dfe75e3e44cdedbd3d516b8db7fb31932cac7e66d9abfed7a0624cf6042a1a8d"}
Nov 01 20:41:20.789052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.788968    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:20.790628 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.790602    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2" event={"ID":"7a573777-e7f2-4616-8857-06c70e61e5c2","Type":"ContainerStarted","Data":"1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0"}
Nov 01 20:41:20.790775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.790680    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2" event={"ID":"7a573777-e7f2-4616-8857-06c70e61e5c2","Type":"ContainerStarted","Data":"5d705fe018fe183dd05c318daca7b5edcb8de787adbc72cb89deecf1cb779998"}
Nov 01 20:41:20.790775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.790746    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:20.804840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.804774    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx" podStartSLOduration=1.245972218 podStartE2EDuration="1.804756713s" totalImagesPullingTime="558.784495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:20.803973919 +0000 UTC m=+2526.219606392" watchObservedRunningTime="2026-11-01 20:41:20.804756713 +0000 UTC m=+2526.220389188"
Nov 01 20:41:20.819433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.819384    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg" podStartSLOduration=1.819361318 podStartE2EDuration="1.819361318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:20.81874474 +0000 UTC m=+2526.234377214" watchObservedRunningTime="2026-11-01 20:41:20.819361318 +0000 UTC m=+2526.234993793"
Nov 01 20:41:20.835338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.835279    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2" podStartSLOduration=1.835267812 podStartE2EDuration="1.835267812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:20.832847739 +0000 UTC m=+2526.248480213" watchObservedRunningTime="2026-11-01 20:41:20.835267812 +0000 UTC m=+2526.250900286"
Nov 01 20:41:20.859523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:20.859484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:21.221788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:21.221751    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:41:21.642863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:21.642784    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:22.260022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:22.259983    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:22.845676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:22.845616    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:23.449577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:23.449540    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:23.548773 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:23.548738    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:24.052847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:24.052812    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:24.749546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:24.749509    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:25.440492 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:25.440459    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:26.122479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:26.122443    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:26.736195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:26.736159    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:26.807556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:26.807524    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:27.353455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:27.353419    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:28.010206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:28.010166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:28.730034 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:28.729993    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:29.191421 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:29.191329    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:41:29.447967 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:29.447932    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:30.129149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:30.129118    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:30.766671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:30.764240    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:31.345984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:31.345929    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:31.799928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:31.799899    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:31.800101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:31.799968    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:32.077526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:32.077436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:32.662870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:32.662826    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:33.263668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:33.258089    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:33.860416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:33.860303    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:34.646521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:34.646481    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:35.260067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:35.260032    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:35.850058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:35.849997    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:36.451328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.451293    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:36.886077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.886045    6479 generic.go:356] "Generic (PLEG): container finished" podID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerID="cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237" exitCode=0
Nov 01 20:41:36.886345 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.886123    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" event={"ID":"02460fc7-dd51-42a7-8e6d-dde799a5c428","Type":"ContainerDied","Data":"cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"}
Nov 01 20:41:36.886592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.886575    6479 scope.go:122] "RemoveContainer" containerID="cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"
Nov 01 20:41:36.887810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.887780    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:41:36.887914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.887880    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:41:36.887960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:36.887945    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:41:37.063954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:37.063826    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:37.676883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:37.674289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:37.893084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:37.893047    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" event={"ID":"02460fc7-dd51-42a7-8e6d-dde799a5c428","Type":"ContainerStarted","Data":"e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57"}
Nov 01 20:41:38.262672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:38.257220    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:38.342538 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:38.342492    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:38.342926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:38.342881    6479 patch_prober.go:29] interesting pod/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 01 20:41:38.343056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:38.342947    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 01 20:41:38.868668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:38.867712    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:39.342056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:39.341989    6479 patch_prober.go:29] interesting pod/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 01 20:41:39.342056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:39.342039    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 01 20:41:39.566697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:39.566547    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:40.163230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:40.163172    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:40.341924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:40.341891    6479 patch_prober.go:29] interesting pod/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 01 20:41:40.342143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:40.341938    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 01 20:41:40.741897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:40.741829    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:41.342796 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:41.342760    6479 patch_prober.go:29] interesting pod/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 01 20:41:41.343313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:41.342814    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 01 20:41:41.370722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:41.370599    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:41.949668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:41.949604    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:42.342110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:42.342023    6479 patch_prober.go:29] interesting pod/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 01 20:41:42.342110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:42.342087    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 01 20:41:42.542498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:42.542465    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:43.243924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:43.243885    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:43.346015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:43.345983    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:43.346324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:43.346300    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:43.347006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:43.346986    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:44.049008 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:44.048967    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:44.086558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:44.086523    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"]
Nov 01 20:41:44.099128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:44.099094    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"]
Nov 01 20:41:44.148731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:44.148687    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:44.249544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:44.249508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:44.621336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:44.621287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:45.079991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.079950    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"]
Nov 01 20:41:45.080510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.080167    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx" podUID="acce6506-4fa8-4c44-afee-a94a9e6b7161" containerName="maas-api" containerID="cri-o://e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f" gracePeriod=30
Nov 01 20:41:45.121350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.121310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:45.236082 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.236046    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:45.321947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.321907    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:45.379041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.379007    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:45.522607 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.522575    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v\" (UniqueName: \"kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v\") pod \"acce6506-4fa8-4c44-afee-a94a9e6b7161\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") "
Nov 01 20:41:45.522830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.522735    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca\") pod \"acce6506-4fa8-4c44-afee-a94a9e6b7161\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") "
Nov 01 20:41:45.522830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.522769    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs\") pod \"acce6506-4fa8-4c44-afee-a94a9e6b7161\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") "
Nov 01 20:41:45.522830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.522828    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls\") pod \"acce6506-4fa8-4c44-afee-a94a9e6b7161\" (UID: \"acce6506-4fa8-4c44-afee-a94a9e6b7161\") "
Nov 01 20:41:45.523140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.523106    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca" pod "acce6506-4fa8-4c44-afee-a94a9e6b7161" (UID: "acce6506-4fa8-4c44-afee-a94a9e6b7161"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:45.524934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.524897    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs" pod "acce6506-4fa8-4c44-afee-a94a9e6b7161" (UID: "acce6506-4fa8-4c44-afee-a94a9e6b7161"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:45.525050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.524943    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v" pod "acce6506-4fa8-4c44-afee-a94a9e6b7161" (UID: "acce6506-4fa8-4c44-afee-a94a9e6b7161"). InnerVolumeSpecName "kube-api-access-8xj7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:45.525050 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.524985    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls" pod "acce6506-4fa8-4c44-afee-a94a9e6b7161" (UID: "acce6506-4fa8-4c44-afee-a94a9e6b7161"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:45.623742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.623660    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:45.623742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.623693    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xj7v\" (UniqueName: \"kubernetes.io/projected/acce6506-4fa8-4c44-afee-a94a9e6b7161-kube-api-access-8xj7v\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:45.623742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.623705    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce6506-4fa8-4c44-afee-a94a9e6b7161-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:45.623742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.623717    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce6506-4fa8-4c44-afee-a94a9e6b7161-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:45.932532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.932454    6479 generic.go:356] "Generic (PLEG): container finished" podID="acce6506-4fa8-4c44-afee-a94a9e6b7161" containerID="e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f" exitCode=0
Nov 01 20:41:45.932532 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.932502    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"
Nov 01 20:41:45.932771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.932542    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx" event={"ID":"acce6506-4fa8-4c44-afee-a94a9e6b7161","Type":"ContainerDied","Data":"e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f"}
Nov 01 20:41:45.932771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.932597    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx" event={"ID":"acce6506-4fa8-4c44-afee-a94a9e6b7161","Type":"ContainerDied","Data":"2d11eb044c283590a9d570bc89bc65c6c0ec06a8b3a936e3ec176ac4733addd2"}
Nov 01 20:41:45.932771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.932619    6479 scope.go:122] "RemoveContainer" containerID="e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f"
Nov 01 20:41:45.942640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.942623    6479 scope.go:122] "RemoveContainer" containerID="e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f"
Nov 01 20:41:45.942982 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:45.942961    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f\": container with ID starting with e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f not found: ID does not exist" containerID="e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f"
Nov 01 20:41:45.943035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.942993    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f"} err="failed to get container status \"e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f\": rpc error: code = NotFound desc = could not find container \"e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f\": container with ID starting with e52cede9231ad1f3294876502d774afd336b676e3e9d66d0c1586629dcf63d9f not found: ID does not exist"
Nov 01 20:41:45.956086 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.956042    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"]
Nov 01 20:41:45.958067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:45.958034    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-022966d4-7d546b8b8c-mpnzx"]
Nov 01 20:41:46.504877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:46.504837    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:41:46.724987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:46.723898    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acce6506-4fa8-4c44-afee-a94a9e6b7161" path="/var/lib/kubelet/pods/acce6506-4fa8-4c44-afee-a94a9e6b7161/volumes"
Nov 01 20:41:49.087474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.087405    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2" podUID="7a573777-e7f2-4616-8857-06c70e61e5c2" containerName="payload-processing" containerID="cri-o://1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0" gracePeriod=25
Nov 01 20:41:49.100071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.099995    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg" podUID="3bf97345-657f-4940-84aa-66b87d8e931f" containerName="payload-pre-processing" containerID="cri-o://3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95" gracePeriod=25
Nov 01 20:41:49.351565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.351506    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:49.387063 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.387031    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:49.466020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.465977    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p\" (UniqueName: \"kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p\") pod \"7a573777-e7f2-4616-8857-06c70e61e5c2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") "
Nov 01 20:41:49.466178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.466083    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume\") pod \"7a573777-e7f2-4616-8857-06c70e61e5c2\" (UID: \"7a573777-e7f2-4616-8857-06c70e61e5c2\") "
Nov 01 20:41:49.466568 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.466529    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume" pod "7a573777-e7f2-4616-8857-06c70e61e5c2" (UID: "7a573777-e7f2-4616-8857-06c70e61e5c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:49.468069 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.468040    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p" pod "7a573777-e7f2-4616-8857-06c70e61e5c2" (UID: "7a573777-e7f2-4616-8857-06c70e61e5c2"). InnerVolumeSpecName "kube-api-access-l2s4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:49.567015 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.566975    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6\" (UniqueName: \"kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6\") pod \"3bf97345-657f-4940-84aa-66b87d8e931f\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") "
Nov 01 20:41:49.567191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.567116    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume\") pod \"3bf97345-657f-4940-84aa-66b87d8e931f\" (UID: \"3bf97345-657f-4940-84aa-66b87d8e931f\") "
Nov 01 20:41:49.567295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.567282    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2s4p\" (UniqueName: \"kubernetes.io/projected/7a573777-e7f2-4616-8857-06c70e61e5c2-kube-api-access-l2s4p\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:49.567333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.567300    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a573777-e7f2-4616-8857-06c70e61e5c2-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:49.567616 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.567574    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume" pod "3bf97345-657f-4940-84aa-66b87d8e931f" (UID: "3bf97345-657f-4940-84aa-66b87d8e931f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:49.569088 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.569061    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6" pod "3bf97345-657f-4940-84aa-66b87d8e931f" (UID: "3bf97345-657f-4940-84aa-66b87d8e931f"). InnerVolumeSpecName "kube-api-access-sqnv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:49.668210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.668134    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf97345-657f-4940-84aa-66b87d8e931f-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:49.668210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.668162    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqnv6\" (UniqueName: \"kubernetes.io/projected/3bf97345-657f-4940-84aa-66b87d8e931f-kube-api-access-sqnv6\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:49.955018 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.954988    6479 generic.go:356] "Generic (PLEG): container finished" podID="7a573777-e7f2-4616-8857-06c70e61e5c2" containerID="1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0" exitCode=0
Nov 01 20:41:49.955216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.955040    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"
Nov 01 20:41:49.955216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.955066    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2" event={"ID":"7a573777-e7f2-4616-8857-06c70e61e5c2","Type":"ContainerDied","Data":"1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0"}
Nov 01 20:41:49.955216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.955121    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2" event={"ID":"7a573777-e7f2-4616-8857-06c70e61e5c2","Type":"ContainerDied","Data":"5d705fe018fe183dd05c318daca7b5edcb8de787adbc72cb89deecf1cb779998"}
Nov 01 20:41:49.955216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.955144    6479 scope.go:122] "RemoveContainer" containerID="1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0"
Nov 01 20:41:49.956552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.956527    6479 generic.go:356] "Generic (PLEG): container finished" podID="3bf97345-657f-4940-84aa-66b87d8e931f" containerID="3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95" exitCode=0
Nov 01 20:41:49.956710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.956567    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"
Nov 01 20:41:49.956710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.956564    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg" event={"ID":"3bf97345-657f-4940-84aa-66b87d8e931f","Type":"ContainerDied","Data":"3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95"}
Nov 01 20:41:49.956710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.956696    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg" event={"ID":"3bf97345-657f-4940-84aa-66b87d8e931f","Type":"ContainerDied","Data":"dfe75e3e44cdedbd3d516b8db7fb31932cac7e66d9abfed7a0624cf6042a1a8d"}
Nov 01 20:41:49.965041 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.964996    6479 scope.go:122] "RemoveContainer" containerID="1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0"
Nov 01 20:41:49.965304 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:49.965283    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0\": container with ID starting with 1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0 not found: ID does not exist" containerID="1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0"
Nov 01 20:41:49.965367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.965315    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0"} err="failed to get container status \"1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0\": rpc error: code = NotFound desc = could not find container \"1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0\": container with ID starting with 1a12a5d4b462eeb54d46c87d786c22a76488d8523ee742cd0822c9ca73255ea0 not found: ID does not exist"
Nov 01 20:41:49.965367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.965333    6479 scope.go:122] "RemoveContainer" containerID="3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95"
Nov 01 20:41:49.973349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.973331    6479 scope.go:122] "RemoveContainer" containerID="3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95"
Nov 01 20:41:49.973593 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:49.973573    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95\": container with ID starting with 3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95 not found: ID does not exist" containerID="3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95"
Nov 01 20:41:49.973656 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.973603    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95"} err="failed to get container status \"3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95\": rpc error: code = NotFound desc = could not find container \"3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95\": container with ID starting with 3b9dd00e9cb17afcaa344c824415147882c47609f7028b6525a188afee0fda95 not found: ID does not exist"
Nov 01 20:41:49.978817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.978796    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"]
Nov 01 20:41:49.982711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.982691    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-022966d4-977867645-x62z2"]
Nov 01 20:41:49.991992 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.991972    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"]
Nov 01 20:41:49.994337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:49.994316    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-022966d4-76857785ff-tjttg"]
Nov 01 20:41:50.278550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:50.278468    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"]
Nov 01 20:41:50.279021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:50.278753    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy" containerID="cri-o://e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57" gracePeriod=30
Nov 01 20:41:50.306899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:50.306863    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:41:50.706938 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:50.706906    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf97345-657f-4940-84aa-66b87d8e931f" path="/var/lib/kubelet/pods/3bf97345-657f-4940-84aa-66b87d8e931f/volumes"
Nov 01 20:41:50.707294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:50.707281    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a573777-e7f2-4616-8857-06c70e61e5c2" path="/var/lib/kubelet/pods/7a573777-e7f2-4616-8857-06c70e61e5c2/volumes"
Nov 01 20:41:52.104547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:52.104507    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:41:55.714391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.714360    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:55.778370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778326    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778410    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778441    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778471    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778504    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778532    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778588 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778561    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778621    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.778977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.778721    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token\") pod \"02460fc7-dd51-42a7-8e6d-dde799a5c428\" (UID: \"02460fc7-dd51-42a7-8e6d-dde799a5c428\") "
Nov 01 20:41:55.779780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.779739    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:55.780020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.779990    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:55.780278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.780251    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:55.780436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.780418    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:55.780599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.780581    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:55.782556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.782507    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:55.782556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.782531    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "kube-api-access-w8xgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:55.784841 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.784818    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:55.784939 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.784898    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo" pod "02460fc7-dd51-42a7-8e6d-dde799a5c428" (UID: "02460fc7-dd51-42a7-8e6d-dde799a5c428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:41:55.879526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879484    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879514    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879525    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8xgj\" (UniqueName: \"kubernetes.io/projected/02460fc7-dd51-42a7-8e6d-dde799a5c428-kube-api-access-w8xgj\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879537    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02460fc7-dd51-42a7-8e6d-dde799a5c428-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879547    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879557    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879565    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879573    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:55.879957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:55.879581    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02460fc7-dd51-42a7-8e6d-dde799a5c428-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:56.013681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.013620    6479 generic.go:356] "Generic (PLEG): container finished" podID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerID="e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57" exitCode=0
Nov 01 20:41:56.013888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.013690    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"
Nov 01 20:41:56.013888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.013693    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" event={"ID":"02460fc7-dd51-42a7-8e6d-dde799a5c428","Type":"ContainerDied","Data":"e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57"}
Nov 01 20:41:56.013888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.013813    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz" event={"ID":"02460fc7-dd51-42a7-8e6d-dde799a5c428","Type":"ContainerDied","Data":"30e263dd961e7b5d9a1386d29dd206d4a567b9930db3a677f78ea807b01d530d"}
Nov 01 20:41:56.013888 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.013843    6479 scope.go:122] "RemoveContainer" containerID="e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57"
Nov 01 20:41:56.026763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.026722    6479 scope.go:122] "RemoveContainer" containerID="cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"
Nov 01 20:41:56.037415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.037388    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"]
Nov 01 20:41:56.038075 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.038057    6479 scope.go:122] "RemoveContainer" containerID="e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57"
Nov 01 20:41:56.038413 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:56.038384    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57\": container with ID starting with e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57 not found: ID does not exist" containerID="e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57"
Nov 01 20:41:56.038524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.038425    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57"} err="failed to get container status \"e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57\": rpc error: code = NotFound desc = could not find container \"e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57\": container with ID starting with e749646c2c4c4e9c420af05598818ba40f0397a9708badc304f9a5a3a2110a57 not found: ID does not exist"
Nov 01 20:41:56.038524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.038451    6479 scope.go:122] "RemoveContainer" containerID="cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"
Nov 01 20:41:56.038775 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:56.038751    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237\": container with ID starting with cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237 not found: ID does not exist" containerID="cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"
Nov 01 20:41:56.038832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.038785    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237"} err="failed to get container status \"cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237\": rpc error: code = NotFound desc = could not find container \"cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237\": container with ID starting with cbe6d560ae45d6d99ca818379cbf71738222e7f4418f6db61930ffb11460e237 not found: ID does not exist"
Nov 01 20:41:56.042469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.042446    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-022966d4-openshift-default-7795c9cf4c-fvzzz"]
Nov 01 20:41:56.706132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:41:56.706097    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" path="/var/lib/kubelet/pods/02460fc7-dd51-42a7-8e6d-dde799a5c428/volumes"
Nov 01 20:41:59.168929 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:41:59.168888    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:42:01.378857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:01.378823    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:42:01.478592 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:01.478551    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:42:07.733877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:07.733832    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:42:11.477672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:11.467472    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:42:15.210283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:15.210247    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:42:18.153827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:18.153773    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:42:18.154553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:18.154505    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:42:18.175584 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:18.175529    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:42:18.175914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:18.175865    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:42:23.498986 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:23.498945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:42:29.169901 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:42:29.169854    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:42:30.448835 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:30.448792    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:42:35.001814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:35.001759    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-989zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:42:35.002470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:35.002421    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-989zh" podUID="4e8677a8-ebe4-4a37-9a4e-e43551a314ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:42:57.048513 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.048433    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:42:57.310635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.310543    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:42:57.406827 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.406793    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:42:57.747671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.743612    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.837902    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"]
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838461    6479 cpu_manager.go:414] "containerMap: removing container" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838483    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838516    6479 cpu_manager.go:414] "containerMap: removing container" podUID="7a573777-e7f2-4616-8857-06c70e61e5c2" containerName="payload-processing"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838525    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a573777-e7f2-4616-8857-06c70e61e5c2" containerName="payload-processing"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838551    6479 cpu_manager.go:414] "containerMap: removing container" podUID="acce6506-4fa8-4c44-afee-a94a9e6b7161" containerName="maas-api"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838562    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acce6506-4fa8-4c44-afee-a94a9e6b7161" containerName="maas-api"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838583    6479 cpu_manager.go:414] "containerMap: removing container" podUID="3bf97345-657f-4940-84aa-66b87d8e931f" containerName="payload-pre-processing"
Nov 01 20:42:57.839668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.838591    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf97345-657f-4940-84aa-66b87d8e931f" containerName="payload-pre-processing"
Nov 01 20:42:57.840404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.840379    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf97345-657f-4940-84aa-66b87d8e931f" containerName="payload-pre-processing"
Nov 01 20:42:57.840580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.840566    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy"
Nov 01 20:42:57.840705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.840691    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a573777-e7f2-4616-8857-06c70e61e5c2" containerName="payload-processing"
Nov 01 20:42:57.840818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.840806    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy"
Nov 01 20:42:57.840909 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.840899    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="acce6506-4fa8-4c44-afee-a94a9e6b7161" containerName="maas-api"
Nov 01 20:42:57.866186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.866156    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:57.902671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.875768    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"]
Nov 01 20:42:57.930753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.930724    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 20:42:57.930979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.930575    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 20:42:57.937720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.937249    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7zk75\""
Nov 01 20:42:57.937720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.937356    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 20:42:57.951318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.951290    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a7bebb0dadea\""
Nov 01 20:42:57.951593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.951576    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 20:42:57.976463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.961687    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:57.976814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.976784    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:57.977072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.977053    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9t2r\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:57.977281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:57.977263    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.079432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.079318    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.080150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.080108    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.080347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.080323    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.080545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.080526    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9t2r\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.082164 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.082143    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.083848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.083829    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.108489 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.108453    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9t2r\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.152615 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.152577    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials\") pod \"payload-processing-external-model-d88cdf6dc-hn82d\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") " pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.263934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.263881    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:42:58.973670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:58.971507    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"]
Nov 01 20:42:59.462039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:42:59.461902    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d" event={"ID":"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46","Type":"ContainerStarted","Data":"0cc6dbfcb6e06bb618ef570ec60d08a7d509c40995c6b4722bfb45212d25ccdf"}
Nov 01 20:42:59.536172 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:42:59.536128    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:43:00.470928 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:00.470876    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d" event={"ID":"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46","Type":"ContainerStarted","Data":"1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b"}
Nov 01 20:43:00.471595 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:00.471211    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:43:00.489602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:00.489543    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d" podStartSLOduration=3.489525355 podStartE2EDuration="3.489525355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:00.487147689 +0000 UTC m=+2625.902780164" watchObservedRunningTime="2026-11-01 20:43:00.489525355 +0000 UTC m=+2625.905157828"
Nov 01 20:43:02.219670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:02.219516    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:43:03.163836 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:03.163782    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:43:03.165095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:03.165045    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:43:03.174487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:03.174443    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:43:03.174795 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:03.174758    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:43:11.499110 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:11.499074    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:43:14.917411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:14.917338    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:43:23.538288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:43:23.538229    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:43:29.169601 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:43:29.169561    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:43:59.169231 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:43:59.169189    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:44:11.874985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:11.874943    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"]
Nov 01 20:44:11.899788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:11.899750    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:11.973897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:11.973864    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:12.131665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:12.131559    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:44:13.011549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:13.011513    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:13.733222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:13.733170    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:15.835997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:15.835622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:16.140430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:16.140294    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:16.875670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:16.875263    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d" podUID="d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" containerName="payload-processing" containerID="cri-o://1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b" gracePeriod=25
Nov 01 20:44:17.798907 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.798831    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:44:17.920360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.919707    6479 generic.go:356] "Generic (PLEG): container finished" podID="d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" containerID="1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b" exitCode=0
Nov 01 20:44:17.920360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.919771    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d" event={"ID":"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46","Type":"ContainerDied","Data":"1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b"}
Nov 01 20:44:17.920360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.919821    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d" event={"ID":"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46","Type":"ContainerDied","Data":"0cc6dbfcb6e06bb618ef570ec60d08a7d509c40995c6b4722bfb45212d25ccdf"}
Nov 01 20:44:17.920360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.919850    6479 scope.go:122] "RemoveContainer" containerID="1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b"
Nov 01 20:44:17.920360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.920004    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"
Nov 01 20:44:17.934931 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.934897    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r\") pod \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") "
Nov 01 20:44:17.936179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.935146    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials\") pod \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") "
Nov 01 20:44:17.936179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.935195    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume\") pod \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") "
Nov 01 20:44:17.936179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.935256    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume\") pod \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\" (UID: \"d74321ae-83e8-4e9e-90b4-b6d88cf8cf46\") "
Nov 01 20:44:17.936823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.936799    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume" pod "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" (UID: "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:17.938166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.938140    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume" pod "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" (UID: "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:17.938402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.938385    6479 scope.go:122] "RemoveContainer" containerID="1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b"
Nov 01 20:44:17.939558 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:44:17.939372    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b\": container with ID starting with 1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b not found: ID does not exist" containerID="1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b"
Nov 01 20:44:17.939558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.939509    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b"} err="failed to get container status \"1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b\": rpc error: code = NotFound desc = could not find container \"1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b\": container with ID starting with 1261d267139b24aa33a684c4fd7713a032730ced8f725c260d068dfd1b92564b not found: ID does not exist"
Nov 01 20:44:17.943058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.943034    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r" pod "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" (UID: "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46"). InnerVolumeSpecName "kube-api-access-x9t2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:17.945068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:17.945039    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials" pod "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" (UID: "d74321ae-83e8-4e9e-90b4-b6d88cf8cf46"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:18.025392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.025301    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:44:18.037816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.037765    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9t2r\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-kube-api-access-x9t2r\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:18.038111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.038033    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:18.038111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.038060    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:18.038111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.038088    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:18.162203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.161768    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:44:18.162203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.161837    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:44:18.241304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.241271    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"]
Nov 01 20:44:18.244934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.244908    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d88cdf6dc-hn82d"]
Nov 01 20:44:18.709142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:18.709096    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" path="/var/lib/kubelet/pods/d74321ae-83e8-4e9e-90b4-b6d88cf8cf46/volumes"
Nov 01 20:44:19.209542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:19.209504    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:20.802221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:20.801741    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:44:20.802221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:20.801822    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:44:23.574337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:23.574284    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:44:29.169969 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:44:29.169884    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:44:58.617322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:44:58.617287    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:44:59.190890 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:44:59.190817    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:45:17.046753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:17.046715    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:45:17.462978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:17.462945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:45:17.561458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:17.561415    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:45:20.792822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:20.792778    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:45:20.793525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:20.793485    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:45:21.913239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:21.913194    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:45:23.690670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:23.688204    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:45:29.169111 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:45:29.169073    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:45:36.520175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.520136    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"]
Nov 01 20:45:36.520780 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.520758    6479 cpu_manager.go:414] "containerMap: removing container" podUID="d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" containerName="payload-processing"
Nov 01 20:45:36.520856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.520784    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" containerName="payload-processing"
Nov 01 20:45:36.520856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.520817    6479 cpu_manager.go:414] "containerMap: removing container" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy"
Nov 01 20:45:36.520856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.520825    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02460fc7-dd51-42a7-8e6d-dde799a5c428" containerName="istio-proxy"
Nov 01 20:45:36.520973 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.520931    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74321ae-83e8-4e9e-90b4-b6d88cf8cf46" containerName="payload-processing"
Nov 01 20:45:36.524462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.524441    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.531481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.531455    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0d5ad972-kserve-self-signed-certs\""
Nov 01 20:45:36.533760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.533706    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"]
Nov 01 20:45:36.572246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.572204    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:45:36.639359 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.639316    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.639561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.639445    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.639561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.639475    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.639691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.639587    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvd7\" (UniqueName: \"kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.639691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.639629    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.639691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.639669    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.740663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.740619    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvd7\" (UniqueName: \"kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.740880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.740700    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.740880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.740729    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.740880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.740787    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.740880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.740843    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.740880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.740865    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.741258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.741141    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.741258 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.741219    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.741375 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.741294    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.743225 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.743199    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.743485 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.743469    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.748894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.748868    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvd7\" (UniqueName: \"kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7\") pod \"e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") " pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:36.851472 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:36.851382    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:37.128251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:37.128169    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"]
Nov 01 20:45:37.131877 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:45:37.131839    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27a2937b_65b7_4a39_aab9_4a20ba7d0c86.slice/crio-190c9e4d8bcab5b81db6b4ff81ccc7f14038eb0d94ea3fdf1aa66c09d613dc97 WatchSource:0}: Error finding container 190c9e4d8bcab5b81db6b4ff81ccc7f14038eb0d94ea3fdf1aa66c09d613dc97: Status 404 returned error can't find the container with id 190c9e4d8bcab5b81db6b4ff81ccc7f14038eb0d94ea3fdf1aa66c09d613dc97
Nov 01 20:45:37.432469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:37.432370    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj" event={"ID":"27a2937b-65b7-4a39-aab9-4a20ba7d0c86","Type":"ContainerStarted","Data":"4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e"}
Nov 01 20:45:37.432469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:37.432429    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj" event={"ID":"27a2937b-65b7-4a39-aab9-4a20ba7d0c86","Type":"ContainerStarted","Data":"190c9e4d8bcab5b81db6b4ff81ccc7f14038eb0d94ea3fdf1aa66c09d613dc97"}
Nov 01 20:45:37.448241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:37.448169    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj" podStartSLOduration=1.448154846 podStartE2EDuration="1.448154846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:37.448017736 +0000 UTC m=+2782.863650208" watchObservedRunningTime="2026-11-01 20:45:37.448154846 +0000 UTC m=+2782.863787320"
Nov 01 20:45:37.852095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:37.852053    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:37.917238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:37.917197    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:45:46.852006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:46.851961    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:46.864516 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:46.864478    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:47.529529 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:47.529496    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:45:52.972216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:52.972184    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"]
Nov 01 20:45:52.975964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:52.975924    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:52.985058 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:52.985024    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0d5ad972-kserve-self-signed-certs\""
Nov 01 20:45:52.987304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:52.987272    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"]
Nov 01 20:45:53.048000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.047967    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.048196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.048007    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.048196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.048089    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.048196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.048153    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.048196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.048175    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.048356 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.048219    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njfg\" (UniqueName: \"kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.149849 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.149803    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.150205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.150171    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.150810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.150781    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.151418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.151398    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.151885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.151862    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.152024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.151920    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njfg\" (UniqueName: \"kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.152024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.151796    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.152187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.151272    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.152681 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.150346    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.156320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.156281    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.156727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.156706    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.162384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.162359    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njfg\" (UniqueName: \"kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg\") pod \"e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") " pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.163639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.163614    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:45:53.294822 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.294726    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:53.568933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:53.568752    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"]
Nov 01 20:45:53.571539 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:45:53.571508    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01ee460c_c7cd_40f9_93d8_741fd3fb2f88.slice/crio-1f1b3c528ec3bc2ae84e2cd7236cd7424253bdce4350c95b56c64fc26507e5da WatchSource:0}: Error finding container 1f1b3c528ec3bc2ae84e2cd7236cd7424253bdce4350c95b56c64fc26507e5da: Status 404 returned error can't find the container with id 1f1b3c528ec3bc2ae84e2cd7236cd7424253bdce4350c95b56c64fc26507e5da
Nov 01 20:45:54.554596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:54.554563    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" event={"ID":"01ee460c-c7cd-40f9-93d8-741fd3fb2f88","Type":"ContainerStarted","Data":"3bfe6a2318f6ecc4c5860bfc8ca55bf222f32ba788656ac51e2ca1ed79d9520f"}
Nov 01 20:45:54.554596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:54.554603    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" event={"ID":"01ee460c-c7cd-40f9-93d8-741fd3fb2f88","Type":"ContainerStarted","Data":"1f1b3c528ec3bc2ae84e2cd7236cd7424253bdce4350c95b56c64fc26507e5da"}
Nov 01 20:45:54.571360 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:54.571309    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" podStartSLOduration=2.571295057 podStartE2EDuration="2.571295057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:54.569274311 +0000 UTC m=+2799.984906786" watchObservedRunningTime="2026-11-01 20:45:54.571295057 +0000 UTC m=+2799.986927529"
Nov 01 20:45:54.635132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:54.635094    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:45:55.295713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:45:55.295641    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:45:59.170566 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:45:59.170520    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:46:03.295913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:03.295869    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:46:03.308305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:03.308275    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:46:03.626530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:03.626453    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:46:09.641534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:09.641492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:10.336443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:10.336404    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:46:20.740684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:20.739878    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"]
Nov 01 20:46:20.779686 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:20.779622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:22.008073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:22.008031    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:46:22.392519 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:22.392411    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"]
Nov 01 20:46:22.435609 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:22.435486    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:23.795683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:23.795181    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:27.756710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:27.756671    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:46:29.220592 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:46:29.220470    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:46:35.761985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:35.761942    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj" podUID="27a2937b-65b7-4a39-aab9-4a20ba7d0c86" containerName="main" containerID="cri-o://4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e" gracePeriod=45
Nov 01 20:46:36.011023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.010998    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:46:36.124964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.124873    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir\") pod \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") "
Nov 01 20:46:36.124964 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.124924    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home\") pod \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") "
Nov 01 20:46:36.125191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.124977    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache\") pod \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") "
Nov 01 20:46:36.125191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125015    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7\" (UniqueName: \"kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7\") pod \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") "
Nov 01 20:46:36.125191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125099    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs\" (UniqueName: \"kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs\") pod \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") "
Nov 01 20:46:36.125372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125197    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm\") pod \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\" (UID: \"27a2937b-65b7-4a39-aab9-4a20ba7d0c86\") "
Nov 01 20:46:36.125372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125214    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir" pod "27a2937b-65b7-4a39-aab9-4a20ba7d0c86" (UID: "27a2937b-65b7-4a39-aab9-4a20ba7d0c86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:36.125372 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125231    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home" pod "27a2937b-65b7-4a39-aab9-4a20ba7d0c86" (UID: "27a2937b-65b7-4a39-aab9-4a20ba7d0c86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:36.125528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125400    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache" pod "27a2937b-65b7-4a39-aab9-4a20ba7d0c86" (UID: "27a2937b-65b7-4a39-aab9-4a20ba7d0c86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:36.125528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125518    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:36.125610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125535    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:36.125610 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.125548    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:36.127410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.127376    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs" pod "27a2937b-65b7-4a39-aab9-4a20ba7d0c86" (UID: "27a2937b-65b7-4a39-aab9-4a20ba7d0c86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:46:36.127410 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.127403    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7" pod "27a2937b-65b7-4a39-aab9-4a20ba7d0c86" (UID: "27a2937b-65b7-4a39-aab9-4a20ba7d0c86"). InnerVolumeSpecName "kube-api-access-vcvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:36.127633 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.127615    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm" pod "27a2937b-65b7-4a39-aab9-4a20ba7d0c86" (UID: "27a2937b-65b7-4a39-aab9-4a20ba7d0c86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:36.226832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.226793    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:36.226832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.226830    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcvd7\" (UniqueName: \"kubernetes.io/projected/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-kube-api-access-vcvd7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:36.227056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.226843    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a2937b-65b7-4a39-aab9-4a20ba7d0c86-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:36.537683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.537635    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:46:36.864619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.864438    6479 generic.go:356] "Generic (PLEG): container finished" podID="27a2937b-65b7-4a39-aab9-4a20ba7d0c86" containerID="4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e" exitCode=0
Nov 01 20:46:36.864619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.864488    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj" event={"ID":"27a2937b-65b7-4a39-aab9-4a20ba7d0c86","Type":"ContainerDied","Data":"4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e"}
Nov 01 20:46:36.864619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.864532    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"
Nov 01 20:46:36.864619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.864549    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj" event={"ID":"27a2937b-65b7-4a39-aab9-4a20ba7d0c86","Type":"ContainerDied","Data":"190c9e4d8bcab5b81db6b4ff81ccc7f14038eb0d94ea3fdf1aa66c09d613dc97"}
Nov 01 20:46:36.864619 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.864574    6479 scope.go:122] "RemoveContainer" containerID="4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e"
Nov 01 20:46:36.879680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.874672    6479 scope.go:122] "RemoveContainer" containerID="4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e"
Nov 01 20:46:36.879680 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:46:36.874999    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e\": container with ID starting with 4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e not found: ID does not exist" containerID="4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e"
Nov 01 20:46:36.879680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.875046    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e"} err="failed to get container status \"4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e\": rpc error: code = NotFound desc = could not find container \"4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e\": container with ID starting with 4dea7bd11e60b3ee539ee4ac1e2bba7627e9450bd48e4b6c7cc68b206bcf9d2e not found: ID does not exist"
Nov 01 20:46:36.880913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.880891    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"]
Nov 01 20:46:36.884405 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:36.884379    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0d5ad972-kserve-55f9555c5c-s9xdj"]
Nov 01 20:46:37.437896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:37.437827    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" podUID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" containerName="main" containerID="cri-o://3bfe6a2318f6ecc4c5860bfc8ca55bf222f32ba788656ac51e2ca1ed79d9520f" gracePeriod=45
Nov 01 20:46:37.618112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:37.618076    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" podUID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.198:8000/ready\": dial tcp 10.134.0.198:8000: connect: connection refused"
Nov 01 20:46:37.886134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:37.886096    6479 generic.go:356] "Generic (PLEG): container finished" podID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" containerID="3bfe6a2318f6ecc4c5860bfc8ca55bf222f32ba788656ac51e2ca1ed79d9520f" exitCode=0
Nov 01 20:46:37.890805 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:37.889757    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" event={"ID":"01ee460c-c7cd-40f9-93d8-741fd3fb2f88","Type":"ContainerDied","Data":"3bfe6a2318f6ecc4c5860bfc8ca55bf222f32ba788656ac51e2ca1ed79d9520f"}
Nov 01 20:46:38.561677 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.559048    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:46:38.681729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.677935    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir\") pod \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") "
Nov 01 20:46:38.685673 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.682013    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache\") pod \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") "
Nov 01 20:46:38.685904 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.685881    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm\") pod \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") "
Nov 01 20:46:38.690826 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.690800    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs\") pod \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") "
Nov 01 20:46:38.695729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.695703    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg\" (UniqueName: \"kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg\") pod \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") "
Nov 01 20:46:38.695900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.695884    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home\") pod \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\" (UID: \"01ee460c-c7cd-40f9-93d8-741fd3fb2f88\") "
Nov 01 20:46:38.704799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.704761    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir" pod "01ee460c-c7cd-40f9-93d8-741fd3fb2f88" (UID: "01ee460c-c7cd-40f9-93d8-741fd3fb2f88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:38.713769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.713735    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home" pod "01ee460c-c7cd-40f9-93d8-741fd3fb2f88" (UID: "01ee460c-c7cd-40f9-93d8-741fd3fb2f88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:38.754887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.742871    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs" pod "01ee460c-c7cd-40f9-93d8-741fd3fb2f88" (UID: "01ee460c-c7cd-40f9-93d8-741fd3fb2f88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:46:38.754887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.744767    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache" pod "01ee460c-c7cd-40f9-93d8-741fd3fb2f88" (UID: "01ee460c-c7cd-40f9-93d8-741fd3fb2f88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:38.754887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.744954    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm" pod "01ee460c-c7cd-40f9-93d8-741fd3fb2f88" (UID: "01ee460c-c7cd-40f9-93d8-741fd3fb2f88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:46:38.754887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.751845    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a2937b-65b7-4a39-aab9-4a20ba7d0c86" path="/var/lib/kubelet/pods/27a2937b-65b7-4a39-aab9-4a20ba7d0c86/volumes"
Nov 01 20:46:38.765419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.758822    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg" pod "01ee460c-c7cd-40f9-93d8-741fd3fb2f88" (UID: "01ee460c-c7cd-40f9-93d8-741fd3fb2f88"). InnerVolumeSpecName "kube-api-access-9njfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:38.800445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.800282    6479 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-model-cache\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:38.800445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.800327    6479 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-dshm\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:38.800445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.800343    6479 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tls-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:38.800445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.800358    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njfg\" (UniqueName: \"kubernetes.io/projected/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-kube-api-access-9njfg\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:38.800445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.800374    6479 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-home\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:38.800445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.800418    6479 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ee460c-c7cd-40f9-93d8-741fd3fb2f88-tmp-dir\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:38.925246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.925210    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp" event={"ID":"01ee460c-c7cd-40f9-93d8-741fd3fb2f88","Type":"ContainerDied","Data":"1f1b3c528ec3bc2ae84e2cd7236cd7424253bdce4350c95b56c64fc26507e5da"}
Nov 01 20:46:38.925979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.925955    6479 scope.go:122] "RemoveContainer" containerID="3bfe6a2318f6ecc4c5860bfc8ca55bf222f32ba788656ac51e2ca1ed79d9520f"
Nov 01 20:46:38.926208 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.925894    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"
Nov 01 20:46:38.976093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.976062    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"]
Nov 01 20:46:38.978490 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:38.978465    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0d5ad972-kserve-76df95df97-qfktp"]
Nov 01 20:46:40.723672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:40.723256    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" path="/var/lib/kubelet/pods/01ee460c-c7cd-40f9-93d8-741fd3fb2f88/volumes"
Nov 01 20:46:41.356680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:41.355566    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:46:54.305734 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:54.305693    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:46:54.821897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:54.821861    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:55.180530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.180446    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"]
Nov 01 20:46:55.180883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.180866    6479 cpu_manager.go:414] "containerMap: removing container" podUID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" containerName="main"
Nov 01 20:46:55.180883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.180882    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" containerName="main"
Nov 01 20:46:55.181048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.180907    6479 cpu_manager.go:414] "containerMap: removing container" podUID="27a2937b-65b7-4a39-aab9-4a20ba7d0c86" containerName="main"
Nov 01 20:46:55.181048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.180914    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a2937b-65b7-4a39-aab9-4a20ba7d0c86" containerName="main"
Nov 01 20:46:55.181048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.180997    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ee460c-c7cd-40f9-93d8-741fd3fb2f88" containerName="main"
Nov 01 20:46:55.181048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.181009    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a2937b-65b7-4a39-aab9-4a20ba7d0c86" containerName="main"
Nov 01 20:46:55.184198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.184175    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.193763 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.193735    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c268d1-openshift-default-dockercfg-wb97t\""
Nov 01 20:46:55.197269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.197245    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"]
Nov 01 20:46:55.213527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213486    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213564    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213587    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213604    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213690 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213677    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213717    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzsn\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213733    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213765    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.213876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.213786    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.320366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.320325    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.321546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.321516    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.327463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.323030    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.327715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.327432    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328007    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328176    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328210    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328286    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328355    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzsn\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328383    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328394    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.328509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.328138    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.330013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.329988    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.330132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.330018    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.333710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.333685    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.335039 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.334983    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.341056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.341032    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzsn\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.351190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.351162    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token\") pod \"e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") " pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.500389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.500351    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:55.779550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.779513    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"]
Nov 01 20:46:55.788855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.788162    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:46:55.788855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.788253    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:46:55.788855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:55.788291    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:46:56.032271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:56.032182    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" event={"ID":"96f33907-3c67-4187-923a-5f7c4e75b31e","Type":"ContainerStarted","Data":"5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"}
Nov 01 20:46:56.032271 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:56.032246    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" event={"ID":"96f33907-3c67-4187-923a-5f7c4e75b31e","Type":"ContainerStarted","Data":"5a730716a66ac103b734e05a711049de87fbd5c094a5addd1c6398dfcd5f0631"}
Nov 01 20:46:56.051060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:56.051014    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podStartSLOduration=1.050999523 podStartE2EDuration="1.050999523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:56.049215008 +0000 UTC m=+2861.464847480" watchObservedRunningTime="2026-11-01 20:46:56.050999523 +0000 UTC m=+2861.466631992"
Nov 01 20:46:56.469342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:56.469301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:56.500553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:56.500515    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:56.505866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:56.505839    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:57.036965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:57.036932    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:57.037994 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:57.037968    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:46:58.167128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:58.167088    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:58.267237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:58.267195    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:59.169118 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:46:59.169083    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:46:59.324918 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.324882    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"]
Nov 01 20:46:59.329235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.329211    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.336294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.336267    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c268d1-serving-cert\""
Nov 01 20:46:59.339019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.338996    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"]
Nov 01 20:46:59.354309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.354086    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:59.365302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.365271    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.365534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.365325    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.365534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.365445    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.365534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.365494    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ln46\" (UniqueName: \"kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.384425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.384394    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"]
Nov 01 20:46:59.388201 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.388175    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.396084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.396058    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"]
Nov 01 20:46:59.399118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.399092    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c268d1\""
Nov 01 20:46:59.400895 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.400875    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"]
Nov 01 20:46:59.404563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.404542    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.412226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.412200    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"]
Nov 01 20:46:59.437005 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.436938    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c268d1-dockercfg-hkm4s\""
Nov 01 20:46:59.462321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.462288    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:59.466981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.466959    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxhp\" (UniqueName: \"kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.467060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.466997    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.467060 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467018    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.467144 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:46:59.467125    6479 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-serving-cert: secret "maas-api-e2e-ipp-a-c268d1-serving-cert" not found
Nov 01 20:46:59.467195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467182    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5th\" (UniqueName: \"kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th\") pod \"payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.467232 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:46:59.467203    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls podName:f96002ef-603e-4812-99b1-21ca3009ee8e nodeName:}" failed. No retries permitted until 2026-11-01 20:46:59.967184182 +0000 UTC m=+2865.382816633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls") pod "maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" (UID: "f96002ef-603e-4812-99b1-21ca3009ee8e") : secret "maas-api-e2e-ipp-a-c268d1-serving-cert" not found
Nov 01 20:46:59.467278 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467252    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.467414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467391    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.467480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467442    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.467480 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467472    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ln46\" (UniqueName: \"kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.467718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.467697    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.469813 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.469793    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.478081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.478055    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ln46\" (UniqueName: \"kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:46:59.569237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.569194    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxhp\" (UniqueName: \"kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.569461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.569288    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5th\" (UniqueName: \"kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th\") pod \"payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.569461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.569323    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.569461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.569425    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.570255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.570231    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.575665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.571467    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.580054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.580025    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5th\" (UniqueName: \"kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th\") pod \"payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.581661 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.581619    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxhp\" (UniqueName: \"kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.703945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.703910    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:46:59.722064 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.722026    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:46:59.776683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.776546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:46:59.999036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:46:59.998935    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:47:00.001100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.001072    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"]
Nov 01 20:47:00.007474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.007240    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:47:00.032120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.032017    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"]
Nov 01 20:47:00.044445 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:47:00.044413    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a86150_06ee_417e_9339_c1a8108392b1.slice/crio-151d6096f758ad0bb2853eae2cbacaa4ac7ffdddcb91fcd393408cea1eede2eb WatchSource:0}: Error finding container 151d6096f758ad0bb2853eae2cbacaa4ac7ffdddcb91fcd393408cea1eede2eb: Status 404 returned error can't find the container with id 151d6096f758ad0bb2853eae2cbacaa4ac7ffdddcb91fcd393408cea1eede2eb
Nov 01 20:47:00.053574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.053539    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq" event={"ID":"bd167fb7-953a-48b4-819f-50157fd0f7ca","Type":"ContainerStarted","Data":"d5176c037a134b0982f58fd766387ff366c4d8bed7f6aaedcbb01cded1be6978"}
Nov 01 20:47:00.054868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.054842    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz" event={"ID":"63a86150-06ee-417e-9339-c1a8108392b1","Type":"ContainerStarted","Data":"151d6096f758ad0bb2853eae2cbacaa4ac7ffdddcb91fcd393408cea1eede2eb"}
Nov 01 20:47:00.059114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.059091    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:00.245188 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.245159    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:47:00.521975 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.521943    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"]
Nov 01 20:47:00.523849 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:47:00.523800    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96002ef_603e_4812_99b1_21ca3009ee8e.slice/crio-477c48ba7906311670a9b94a65c963f515fca56da4ea88a0342aa4ed25efe58d WatchSource:0}: Error finding container 477c48ba7906311670a9b94a65c963f515fca56da4ea88a0342aa4ed25efe58d: Status 404 returned error can't find the container with id 477c48ba7906311670a9b94a65c963f515fca56da4ea88a0342aa4ed25efe58d
Nov 01 20:47:00.525834 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.525805    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:47:00.777382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:00.777292    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:01.060393 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.060359    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" event={"ID":"f96002ef-603e-4812-99b1-21ca3009ee8e","Type":"ContainerStarted","Data":"477c48ba7906311670a9b94a65c963f515fca56da4ea88a0342aa4ed25efe58d"}
Nov 01 20:47:01.062001 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.061973    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz" event={"ID":"63a86150-06ee-417e-9339-c1a8108392b1","Type":"ContainerStarted","Data":"ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094"}
Nov 01 20:47:01.062134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.062094    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:47:01.063618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.063568    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq" event={"ID":"bd167fb7-953a-48b4-819f-50157fd0f7ca","Type":"ContainerStarted","Data":"fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec"}
Nov 01 20:47:01.063762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.063693    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:47:01.081275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.081219    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz" podStartSLOduration=2.081204488 podStartE2EDuration="2.081204488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:01.080103121 +0000 UTC m=+2866.495735597" watchObservedRunningTime="2026-11-01 20:47:01.081204488 +0000 UTC m=+2866.496836960"
Nov 01 20:47:01.095714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.095627    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq" podStartSLOduration=2.095609407 podStartE2EDuration="2.095609407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:01.094247303 +0000 UTC m=+2866.509879777" watchObservedRunningTime="2026-11-01 20:47:01.095609407 +0000 UTC m=+2866.511241881"
Nov 01 20:47:01.221159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.221125    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:47:01.424406 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:01.424308    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:02.071390 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:02.071344    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" event={"ID":"f96002ef-603e-4812-99b1-21ca3009ee8e","Type":"ContainerStarted","Data":"87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f"}
Nov 01 20:47:02.071733 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:02.071708    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:47:02.076590 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:02.076563    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:02.090605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:02.090544    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" podStartSLOduration=2.561825848 podStartE2EDuration="3.090533496s" totalImagesPullingTime="528.707648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:02.088335967 +0000 UTC m=+2867.503968440" watchObservedRunningTime="2026-11-01 20:47:02.090533496 +0000 UTC m=+2867.506165963"
Nov 01 20:47:02.659671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:02.657095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:03.329626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:03.329579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:03.943983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:03.943942    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:04.550461 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:04.550425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:05.142242 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:05.142199    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:05.773672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:05.766848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:06.347456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:06.347413    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:06.942966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:06.942931    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:07.567671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:07.567477    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:08.088120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:08.088090    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:47:08.152065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:08.152031    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:08.758935 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:08.758891    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:09.136971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:09.136891    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:09.237177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:09.237140    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:09.456374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:09.456344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:10.181114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:10.181056    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:10.752251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:10.752213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:11.369941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:11.369869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:11.945979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:11.945945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:12.038991 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.038956    6479 patch_prober.go:29] interesting pod/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 01 20:47:12.039170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.039025    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 01 20:47:12.074019 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.073991    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:47:12.074219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.074057    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:47:12.135044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.135013    6479 generic.go:356] "Generic (PLEG): container finished" podID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerID="5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d" exitCode=0
Nov 01 20:47:12.135214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.135089    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" event={"ID":"96f33907-3c67-4187-923a-5f7c4e75b31e","Type":"ContainerDied","Data":"5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"}
Nov 01 20:47:12.135603 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.135584    6479 scope.go:122] "RemoveContainer" containerID="5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"
Nov 01 20:47:12.137007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.136974    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:12.137107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.137057    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:12.137153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.137104    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:12.548659 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:12.548535    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:13.079084 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.079050    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"]
Nov 01 20:47:13.080342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.080317    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"]
Nov 01 20:47:13.133959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.133920    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:13.142425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.142386    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" event={"ID":"96f33907-3c67-4187-923a-5f7c4e75b31e","Type":"ContainerStarted","Data":"929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa"}
Nov 01 20:47:13.233675 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.233618    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:13.392073 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.391925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:13.501508 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.501474    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:47:13.501775 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.501741    6479 patch_prober.go:29] interesting pod/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 01 20:47:13.501845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.501785    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 01 20:47:13.751229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:13.751189    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:14.471534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:14.471500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:14.501536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:14.501503    6479 patch_prober.go:29] interesting pod/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 01 20:47:14.501717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:14.501557    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 01 20:47:14.900803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:14.900696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:14.982742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:14.982705    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:15.350020 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:15.349985    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:15.501401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:15.501363    6479 patch_prober.go:29] interesting pod/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 01 20:47:15.501576 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:15.501416    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 01 20:47:15.967176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:15.967141    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:16.500997 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:16.500959    6479 patch_prober.go:29] interesting pod/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 01 20:47:16.501186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:16.501019    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 01 20:47:16.748554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:16.748515    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:17.422002 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:17.421909    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:17.501548 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:17.501510    6479 patch_prober.go:29] interesting pod/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 01 20:47:17.501881 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:17.501850    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 01 20:47:18.079712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.079627    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq" podUID="bd167fb7-953a-48b4-819f-50157fd0f7ca" containerName="payload-processing" containerID="cri-o://fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec" gracePeriod=25
Nov 01 20:47:18.081180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.081135    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz" podUID="63a86150-06ee-417e-9339-c1a8108392b1" containerName="payload-pre-processing" containerID="cri-o://ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094" gracePeriod=25
Nov 01 20:47:18.158237 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.158194    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:18.399078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.399051    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:47:18.403067 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.403041    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:47:18.435878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.435840    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume\") pod \"63a86150-06ee-417e-9339-c1a8108392b1\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") "
Nov 01 20:47:18.436353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.435945    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp\" (UniqueName: \"kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp\") pod \"63a86150-06ee-417e-9339-c1a8108392b1\" (UID: \"63a86150-06ee-417e-9339-c1a8108392b1\") "
Nov 01 20:47:18.436353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.435975    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume\") pod \"bd167fb7-953a-48b4-819f-50157fd0f7ca\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") "
Nov 01 20:47:18.436353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.436026    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th\" (UniqueName: \"kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th\") pod \"bd167fb7-953a-48b4-819f-50157fd0f7ca\" (UID: \"bd167fb7-953a-48b4-819f-50157fd0f7ca\") "
Nov 01 20:47:18.437749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.437668    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume" pod "bd167fb7-953a-48b4-819f-50157fd0f7ca" (UID: "bd167fb7-953a-48b4-819f-50157fd0f7ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:47:18.437749 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.437708    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume" pod "63a86150-06ee-417e-9339-c1a8108392b1" (UID: "63a86150-06ee-417e-9339-c1a8108392b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:47:18.439627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.439599    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp" pod "63a86150-06ee-417e-9339-c1a8108392b1" (UID: "63a86150-06ee-417e-9339-c1a8108392b1"). InnerVolumeSpecName "kube-api-access-qdxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:47:18.439914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.439851    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th" pod "bd167fb7-953a-48b4-819f-50157fd0f7ca" (UID: "bd167fb7-953a-48b4-819f-50157fd0f7ca"). InnerVolumeSpecName "kube-api-access-dt5th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:47:18.504981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.504950    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:47:18.505262 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.505228    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:47:18.505941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.505923    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:47:18.537829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.537790    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a86150-06ee-417e-9339-c1a8108392b1-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:47:18.537829 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.537829    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxhp\" (UniqueName: \"kubernetes.io/projected/63a86150-06ee-417e-9339-c1a8108392b1-kube-api-access-qdxhp\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:47:18.538046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.537845    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd167fb7-953a-48b4-819f-50157fd0f7ca-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:47:18.538046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.537860    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5th\" (UniqueName: \"kubernetes.io/projected/bd167fb7-953a-48b4-819f-50157fd0f7ca-kube-api-access-dt5th\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:47:18.765251 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:18.765213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:19.190718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.190680    6479 generic.go:356] "Generic (PLEG): container finished" podID="63a86150-06ee-417e-9339-c1a8108392b1" containerID="ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094" exitCode=0
Nov 01 20:47:19.190887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.190771    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz" event={"ID":"63a86150-06ee-417e-9339-c1a8108392b1","Type":"ContainerDied","Data":"ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094"}
Nov 01 20:47:19.190887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.190779    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"
Nov 01 20:47:19.190887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.190856    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz" event={"ID":"63a86150-06ee-417e-9339-c1a8108392b1","Type":"ContainerDied","Data":"151d6096f758ad0bb2853eae2cbacaa4ac7ffdddcb91fcd393408cea1eede2eb"}
Nov 01 20:47:19.190887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.190885    6479 scope.go:122] "RemoveContainer" containerID="ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094"
Nov 01 20:47:19.192170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.192153    6479 generic.go:356] "Generic (PLEG): container finished" podID="bd167fb7-953a-48b4-819f-50157fd0f7ca" containerID="fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec" exitCode=0
Nov 01 20:47:19.192249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.192207    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"
Nov 01 20:47:19.192303 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.192277    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq" event={"ID":"bd167fb7-953a-48b4-819f-50157fd0f7ca","Type":"ContainerDied","Data":"fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec"}
Nov 01 20:47:19.192350 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.192334    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq" event={"ID":"bd167fb7-953a-48b4-819f-50157fd0f7ca","Type":"ContainerDied","Data":"d5176c037a134b0982f58fd766387ff366c4d8bed7f6aaedcbb01cded1be6978"}
Nov 01 20:47:19.200071 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.200049    6479 scope.go:122] "RemoveContainer" containerID="ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094"
Nov 01 20:47:19.200329 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:47:19.200310    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094\": container with ID starting with ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094 not found: ID does not exist" containerID="ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094"
Nov 01 20:47:19.200385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.200337    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094"} err="failed to get container status \"ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094\": rpc error: code = NotFound desc = could not find container \"ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094\": container with ID starting with ab5b57f05c5a76010251343435e7418dd36afba4464dde4642fe3be2e61b5094 not found: ID does not exist"
Nov 01 20:47:19.200385 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.200353    6479 scope.go:122] "RemoveContainer" containerID="fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec"
Nov 01 20:47:19.210866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.210843    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"]
Nov 01 20:47:19.213797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.213776    6479 scope.go:122] "RemoveContainer" containerID="fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec"
Nov 01 20:47:19.214075 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:47:19.214056    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec\": container with ID starting with fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec not found: ID does not exist" containerID="fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec"
Nov 01 20:47:19.214143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.214083    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec"} err="failed to get container status \"fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec\": rpc error: code = NotFound desc = could not find container \"fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec\": container with ID starting with fac8a6eda04c5c98d679a8a8d22b2b2dba3439f77a4e23e2dfbb6cd46ef4c1ec not found: ID does not exist"
Nov 01 20:47:19.214709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.214640    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-8688f99855-nhshz"]
Nov 01 20:47:19.224263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.224243    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"]
Nov 01 20:47:19.229932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.229902    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-6b88fc9bbf-rtgvq"]
Nov 01 20:47:19.456326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:19.456293    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:20.078432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:20.078395    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:20.706305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:20.706272    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a86150-06ee-417e-9339-c1a8108392b1" path="/var/lib/kubelet/pods/63a86150-06ee-417e-9339-c1a8108392b1/volumes"
Nov 01 20:47:20.706725 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:20.706672    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd167fb7-953a-48b4-819f-50157fd0f7ca" path="/var/lib/kubelet/pods/bd167fb7-953a-48b4-819f-50157fd0f7ca/volumes"
Nov 01 20:47:20.846709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:20.846674    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:21.449953 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:21.449918    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:22.151469 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:22.151438    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:22.756570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:22.756532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:23.474582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:23.474547    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:23.850543 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:23.850462    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:24.064438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:24.064392    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:24.749416 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:24.749380    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:25.361512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:25.361482    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:26.053330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:26.053295    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:26.653409 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:26.653374    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:27.353428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:27.353393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:27.960527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:27.960488    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:28.653710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:28.653673    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:29.169226 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:47:29.169193    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:47:29.364315 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:29.364284    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:30.060333 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:30.060282    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:30.731355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:30.731320    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:31.463754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:31.463725    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:32.076911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:32.076876    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:32.763825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:32.763781    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:33.357890 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:33.357858    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:34.050923 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:34.050891    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:34.652787 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:34.652748    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:34.870140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:34.870101    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:35.249883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.249839    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"]
Nov 01 20:47:35.251170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.251144    6479 cpu_manager.go:414] "containerMap: removing container" podUID="bd167fb7-953a-48b4-819f-50157fd0f7ca" containerName="payload-processing"
Nov 01 20:47:35.251170 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.251167    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd167fb7-953a-48b4-819f-50157fd0f7ca" containerName="payload-processing"
Nov 01 20:47:35.251397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.251192    6479 cpu_manager.go:414] "containerMap: removing container" podUID="63a86150-06ee-417e-9339-c1a8108392b1" containerName="payload-pre-processing"
Nov 01 20:47:35.251397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.251199    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a86150-06ee-417e-9339-c1a8108392b1" containerName="payload-pre-processing"
Nov 01 20:47:35.251397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.251341    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd167fb7-953a-48b4-819f-50157fd0f7ca" containerName="payload-processing"
Nov 01 20:47:35.251397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.251370    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a86150-06ee-417e-9339-c1a8108392b1" containerName="payload-pre-processing"
Nov 01 20:47:35.256526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.256504    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.265778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.265314    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"]
Nov 01 20:47:35.268783 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.268754    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a97e34-openshift-default-dockercfg-fjmzz\""
Nov 01 20:47:35.298302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.298275    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:35.311902 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.311868    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.311904    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.311925    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.312045    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.312113    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.312203    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrwc\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.312251    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.312412    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.312497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.312449    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.394204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.394171    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:35.413774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413739    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.413774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413776    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413795    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413826    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413898    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413949    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrwc\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.413985    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414042    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414068    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414257    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414296    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414640    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414686    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.414936 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.414913    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.416288 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.416253    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.416790 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.416769    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.421976 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.421949    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrwc\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.422459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.422437    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token\") pod \"e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") " pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.494945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.494911    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:35.576845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.576758    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:35.862772 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.862745    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"]
Nov 01 20:47:35.865622 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:47:35.865590    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2a4073_81ad_47b0_86c7_0c7154748c5b.slice/crio-efd1507fd78429e7e8cdb54075df42ea0ceab5aea6e146cb5bba3a58b66d340c WatchSource:0}: Error finding container efd1507fd78429e7e8cdb54075df42ea0ceab5aea6e146cb5bba3a58b66d340c: Status 404 returned error can't find the container with id efd1507fd78429e7e8cdb54075df42ea0ceab5aea6e146cb5bba3a58b66d340c
Nov 01 20:47:35.868206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.868176    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:35.868300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.868239    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:35.868300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:35.868290    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:36.061697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.061657    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:36.277070 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.277031    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" event={"ID":"df2a4073-81ad-47b0-86c7-0c7154748c5b","Type":"ContainerStarted","Data":"751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"}
Nov 01 20:47:36.277697 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.277202    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" event={"ID":"df2a4073-81ad-47b0-86c7-0c7154748c5b","Type":"ContainerStarted","Data":"efd1507fd78429e7e8cdb54075df42ea0ceab5aea6e146cb5bba3a58b66d340c"}
Nov 01 20:47:36.296761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.296717    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podStartSLOduration=1.296699871 podStartE2EDuration="1.296699871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:36.295150289 +0000 UTC m=+2901.710782763" watchObservedRunningTime="2026-11-01 20:47:36.296699871 +0000 UTC m=+2901.712332343"
Nov 01 20:47:36.577534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.577492    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:36.582846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.582815    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:36.761509 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:36.761474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:37.282481 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:37.282440    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:37.283437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:37.283412    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:37.362894 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:37.362857    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:38.151136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:38.151102    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:38.427205 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:38.427117    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:38.528128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:38.528093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:38.912731 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:38.912684    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:39.608916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:39.608870    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:39.700131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:39.699522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:39.874340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:39.874235    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:40.556723 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:40.556684    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:41.287856 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.287816    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"]
Nov 01 20:47:41.293533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.293497    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.302436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.302370    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a97e34-serving-cert\""
Nov 01 20:47:41.304556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.304527    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"]
Nov 01 20:47:41.328047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.328016    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:41.387418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.387385    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.387570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.387506    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.387570 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.387557    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjcc\" (UniqueName: \"kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.387692 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.387592    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.428526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.428474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:41.488482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.488435    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.502919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.502147    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.502919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.502216    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjcc\" (UniqueName: \"kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.502919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.502256    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.503203 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.503061    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.521670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.519954    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.521670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.520894    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.526276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.526235    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjcc\" (UniqueName: \"kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc\") pod \"maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.529785 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.529745    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:41.609196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.609099    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:41.911667 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.908045    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"]
Nov 01 20:47:41.962683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:41.962640    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:42.308525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:42.308488    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh" event={"ID":"8406ce1e-cba2-404e-9a27-1fd30982f7ea","Type":"ContainerStarted","Data":"8d98ffba3ec467c37a63291218120dca262693443d32c55bf7e846e9dd59bd7f"}
Nov 01 20:47:42.648908 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:42.648805    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:43.131474 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:43.131440    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:47:43.315924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:43.315891    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh" event={"ID":"8406ce1e-cba2-404e-9a27-1fd30982f7ea","Type":"ContainerStarted","Data":"3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f"}
Nov 01 20:47:43.316325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:43.316004    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:43.333927 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:43.333879    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh" podStartSLOduration=1.9119678100000002 podStartE2EDuration="2.33386682s" totalImagesPullingTime="421.89901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:43.33210562 +0000 UTC m=+2908.747738093" watchObservedRunningTime="2026-11-01 20:47:43.33386682 +0000 UTC m=+2908.749499293"
Nov 01 20:47:43.445182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:43.445148    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:44.056159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:44.056125    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:44.727006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:44.726972    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:45.466670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:45.458127    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:46.050180 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:46.050143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:46.764219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:46.764155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:47.467670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:47.451826    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:48.066265 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:48.066203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:48.667435 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:48.667379    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:49.331542 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:49.331512    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:47:49.378036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:49.378003    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:50.100153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:50.100123    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:50.848247 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:50.848208    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:51.553357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:51.553319    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:52.073120 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:52.073078    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:52.177331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:52.177291    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:52.862313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:52.862276    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:53.041214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.040184    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": read tcp 10.134.0.2:38920->10.134.0.203:15021: read: connection reset by peer" start-of-body=
Nov 01 20:47:53.041214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.040246    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": read tcp 10.134.0.2:38920->10.134.0.203:15021: read: connection reset by peer"
Nov 01 20:47:53.373940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.373907    6479 generic.go:356] "Generic (PLEG): container finished" podID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerID="751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96" exitCode=0
Nov 01 20:47:53.374142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.373999    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" event={"ID":"df2a4073-81ad-47b0-86c7-0c7154748c5b","Type":"ContainerDied","Data":"751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"}
Nov 01 20:47:53.374630 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.374610    6479 scope.go:122] "RemoveContainer" containerID="751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"
Nov 01 20:47:53.375887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.375852    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:53.375996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.375918    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:53.375996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.375952    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:47:53.444596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.444570    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:53.545297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.545253    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:53.693946 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.693866    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:53.794430 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:53.794279    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:54.387340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:54.387300    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" event={"ID":"df2a4073-81ad-47b0-86c7-0c7154748c5b","Type":"ContainerStarted","Data":"929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017"}
Nov 01 20:47:54.466669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:54.463298    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:54.577095 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:54.577054    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:54.577488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:54.577458    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:47:54.577720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:54.577687    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 01 20:47:55.178672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:55.175213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:55.581081 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:55.581038    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:47:55.581715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:55.581680    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 01 20:47:55.616202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:55.616157    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:55.694583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:55.694541    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:56.123425 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:56.123384    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:56.578160 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:56.578117    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:47:56.578358 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:56.578191    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 01 20:47:56.835580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:56.835500    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:57.564669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:57.562361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:57.577762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:57.577728    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:47:57.578044 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:57.578016    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 01 20:47:58.265148 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:58.265095    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:58.578182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:58.578089    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:47:58.578182 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:58.578158    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 01 20:47:58.877371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:58.877273    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:59.191049 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:47:59.190967    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:47:59.572133 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:59.572089    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:47:59.656722 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:59.656684    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:59.657313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:59.657155    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:47:59.658234 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:47:59.658199    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:48:00.361428 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:00.361396    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:00.971830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:00.971799    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:01.812883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:01.812847    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:02.540240 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:02.540202    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:03.147451 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:03.147416    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:03.858792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:03.858757    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:04.545642 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:04.545608    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:05.153511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:05.153474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:05.851419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:05.851386    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:06.643456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:06.643420    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:07.252657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:07.252623    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:07.955441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:07.955394    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:08.754664 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:08.754620    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:09.356438 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:09.356391    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:09.973924 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:09.973882    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:10.074932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:10.074900    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:10.177176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:10.177136    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:10.752571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:10.752542    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:11.358027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:11.357991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:12.056047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:12.056012    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:12.960627 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:12.960584    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:13.562809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:13.562772    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:14.255215 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:14.255182    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:14.965525 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:14.965491    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:15.564419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:15.564389    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:16.345954 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:16.345914    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:16.671198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:16.671118    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:17.153348 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:17.153308    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:17.958940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:17.958902    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:18.227563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:18.227462    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:48:18.950910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:18.950875    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:19.666596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:19.666554    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:20.279244 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:20.279203    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:21.054363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:21.054326    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:21.759459 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:21.759412    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:22.361971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:22.361931    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:23.056191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:23.056150    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:23.747300 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:23.747265    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:23.887174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:23.887138    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:24.366011 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:24.365973    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:25.054563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:25.054525    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:25.753465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:25.753434    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:26.360305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:26.360271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:27.053378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:27.053344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:27.755026 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:27.754990    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:28.446103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:28.446069    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:29.160432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:29.160397    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:29.169499 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:48:29.169468    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:48:29.928488 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:29.928444    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:30.547392 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:30.547348    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:31.252837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:31.252803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:31.948337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:31.948300    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:32.655214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:32.655177    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:33.361915 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:33.361869    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:34.173900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:34.173859    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:34.980142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:34.980103    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:35.747684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:35.747627    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:36.453246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:36.453214    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:37.062442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:37.062406    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:37.861853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:37.861819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:38.640574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:38.640525    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:39.252329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:39.252277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:39.966847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:39.966800    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:40.657344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:40.657310    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:41.259322 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:41.259286    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:41.963563 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:41.963530    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:42.696958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:42.696921    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:43.350023 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:43.349991    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:44.155699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:44.155663    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:44.645159 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:44.645123    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:48:44.866189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:44.866159    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:45.468597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:45.468568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:46.247530 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:46.247487    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:46.950202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:46.950166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:47.553562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:47.553516    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:48.253142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:48.253107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:48.952204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:48.952166    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:49.558367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:49.558333    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:50.257083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:50.257049    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:50.946947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:50.946915    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:51.555995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:51.555957    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:52.186336 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:52.186293    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:52.284404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:52.284355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:52.955837 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:52.955801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:53.556865 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:53.556828    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:54.271680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:54.271628    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:54.965227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:54.965150    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:55.559750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:55.559709    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:56.260877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:56.260840    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:56.949635 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:56.949594    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:57.251708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:57.251617    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:57.553701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:57.553586    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:58.262079 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:58.262040    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:58.956273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:58.956236    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:48:59.169848 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:48:59.169804    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:48:59.563299 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:48:59.563266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:00.300272 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:00.300230    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:00.964585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:00.964531    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:01.562329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:01.562287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:02.276778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:02.276735    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:03.004351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:03.004302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:03.644455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:03.644393    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:04.359887 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:04.359852    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:05.050209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:05.050173    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:05.651376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:05.651335    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:06.356840 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:06.356806    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:07.050949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:07.050916    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:07.668386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:07.668342    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:08.350114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:08.350068    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:09.084959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:09.084919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:09.653945 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:09.653905    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:10.356331 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:10.356292    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:11.047327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:11.047292    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:11.653152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:11.653117    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:12.353269 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:12.353210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:13.045814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:13.045770    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:13.652940 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:13.652906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:14.361454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:14.361421    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:15.153759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:15.153721    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:15.769926 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:15.769891    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:16.456355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:16.456317    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:17.155555 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:17.155524    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:17.756382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:17.756350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:18.452140 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:18.452107    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:19.147802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:19.147760    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:19.751814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:19.751780    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:20.457932 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:20.457899    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:21.150738 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:21.150693    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:21.761553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:21.761511    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:22.456190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:22.456156    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:23.153730 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:23.153695    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:23.765418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:23.765385    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:23.939970 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:23.939931    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:24.470223 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:24.470190    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:25.158012 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:25.157976    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:25.765147 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:25.765112    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:26.473769 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:26.473731    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:27.208036 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:27.207999    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:27.951572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:27.951537    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:28.659614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:28.659579    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:29.170155 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:49:29.170119    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:49:29.366324 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:29.366289    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:29.955736 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:29.955700    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:30.651253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:30.651216    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:31.444839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:31.444803    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:32.057833 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:32.057788    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:32.857844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:32.857812    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:33.561585 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:33.561544    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:33.945198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:33.945168    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:34.399934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:34.399817    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:34.537384 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:34.537343    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:49:34.651256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:34.651164    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:35.354972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:35.354937    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:36.066455 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:36.066420    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:36.963354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:36.963319    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:37.561114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:37.561081    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:38.257528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:38.257479    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:38.339314 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:38.339279    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:49:38.847389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:38.847350    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:39.564857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:39.564821    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:40.038366 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:40.038332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:40.157720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:40.157677    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:40.851252 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:40.851213    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:41.463727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:41.463687    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:42.155578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:42.155545    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:42.759676 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:42.759622    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:43.470442 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:43.470403    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:44.125777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:44.125744    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:44.840874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:44.840838    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:45.452259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:45.452217    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:46.069418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:46.069374    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:46.247886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:46.247848    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:46.860218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:46.860184    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:47.549306 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:47.549272    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:48.154797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:48.154756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:48.848497 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:48.848454    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:49.310470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.310424    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"]
Nov 01 20:49:49.310955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.310726    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" podUID="f96002ef-603e-4812-99b1-21ca3009ee8e" containerName="maas-api" containerID="cri-o://87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f" gracePeriod=30
Nov 01 20:49:49.339876 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.339844    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:49.418857 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.418818    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:49.669521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.669474    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:49.713068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.713038    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:49:49.841038 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.840996    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca\") pod \"f96002ef-603e-4812-99b1-21ca3009ee8e\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") "
Nov 01 20:49:49.841249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.841051    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46\" (UniqueName: \"kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46\") pod \"f96002ef-603e-4812-99b1-21ca3009ee8e\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") "
Nov 01 20:49:49.841249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.841187    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs\") pod \"f96002ef-603e-4812-99b1-21ca3009ee8e\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") "
Nov 01 20:49:49.841402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.841278    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\") pod \"f96002ef-603e-4812-99b1-21ca3009ee8e\" (UID: \"f96002ef-603e-4812-99b1-21ca3009ee8e\") "
Nov 01 20:49:49.841496 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.841457    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca" pod "f96002ef-603e-4812-99b1-21ca3009ee8e" (UID: "f96002ef-603e-4812-99b1-21ca3009ee8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:49:49.841704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.841683    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96002ef-603e-4812-99b1-21ca3009ee8e-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.843357 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.843322    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls" pod "f96002ef-603e-4812-99b1-21ca3009ee8e" (UID: "f96002ef-603e-4812-99b1-21ca3009ee8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:49:49.843832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.843811    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46" pod "f96002ef-603e-4812-99b1-21ca3009ee8e" (UID: "f96002ef-603e-4812-99b1-21ca3009ee8e"). InnerVolumeSpecName "kube-api-access-5ln46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:49:49.843832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.843819    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs" pod "f96002ef-603e-4812-99b1-21ca3009ee8e" (UID: "f96002ef-603e-4812-99b1-21ca3009ee8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:49:49.943312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.943229    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ln46\" (UniqueName: \"kubernetes.io/projected/f96002ef-603e-4812-99b1-21ca3009ee8e-kube-api-access-5ln46\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.943312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.943260    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.943312 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.943269    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f96002ef-603e-4812-99b1-21ca3009ee8e-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.975919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.975879    6479 generic.go:356] "Generic (PLEG): container finished" podID="f96002ef-603e-4812-99b1-21ca3009ee8e" containerID="87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f" exitCode=0
Nov 01 20:49:49.976101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.975941    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" event={"ID":"f96002ef-603e-4812-99b1-21ca3009ee8e","Type":"ContainerDied","Data":"87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f"}
Nov 01 20:49:49.976101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.975950    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"
Nov 01 20:49:49.976101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.975985    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv" event={"ID":"f96002ef-603e-4812-99b1-21ca3009ee8e","Type":"ContainerDied","Data":"477c48ba7906311670a9b94a65c963f515fca56da4ea88a0342aa4ed25efe58d"}
Nov 01 20:49:49.976101 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.976010    6479 scope.go:122] "RemoveContainer" containerID="87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f"
Nov 01 20:49:49.986861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.986836    6479 scope.go:122] "RemoveContainer" containerID="87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f"
Nov 01 20:49:49.987115 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:49:49.987094    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f\": container with ID starting with 87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f not found: ID does not exist" containerID="87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f"
Nov 01 20:49:49.987218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.987127    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f"} err="failed to get container status \"87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f\": rpc error: code = NotFound desc = could not find container \"87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f\": container with ID starting with 87c698924b54306355ada4a86da15d8583739924727375bf5e21b117a2bdba3f not found: ID does not exist"
Nov 01 20:49:49.999440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:49.999411    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"]
Nov 01 20:49:50.003048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:50.003021    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c268d1-5c479564b5-84tjv"]
Nov 01 20:49:50.365056 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:50.365027    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:50.705704 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:50.705674    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96002ef-603e-4812-99b1-21ca3009ee8e" path="/var/lib/kubelet/pods/f96002ef-603e-4812-99b1-21ca3009ee8e/volumes"
Nov 01 20:49:50.842114 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:50.842076    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:49:50.961042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:50.960945    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:51.657396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:51.657355    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:51.911388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:51.911302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:52.013678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:52.012612    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:52.321186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:52.321143    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:52.421583 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:52.421541    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:52.722886 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:52.722851    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"]
Nov 01 20:49:52.723296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:52.723127    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy" containerID="cri-o://929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa" gracePeriod=30
Nov 01 20:49:52.751227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:52.751190    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:53.248294 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:53.248260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:53.953987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:53.953950    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:54.754567 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:54.754521    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:55.375870 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:55.375782    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:56.054156 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:56.054119    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:56.661007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:56.660947    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:57.357957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:57.357925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:57.963521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:57.963484    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:57.969933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:57.969912    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:49:58.027546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.026608    6479 generic.go:356] "Generic (PLEG): container finished" podID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerID="929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa" exitCode=0
Nov 01 20:49:58.027546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.026738    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" event={"ID":"96f33907-3c67-4187-923a-5f7c4e75b31e","Type":"ContainerDied","Data":"929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa"}
Nov 01 20:49:58.027546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.026857    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl" event={"ID":"96f33907-3c67-4187-923a-5f7c4e75b31e","Type":"ContainerDied","Data":"5a730716a66ac103b734e05a711049de87fbd5c094a5addd1c6398dfcd5f0631"}
Nov 01 20:49:58.027546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.026893    6479 scope.go:122] "RemoveContainer" containerID="929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa"
Nov 01 20:49:58.027546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.027134    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"
Nov 01 20:49:58.054353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.054329    6479 scope.go:122] "RemoveContainer" containerID="5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"
Nov 01 20:49:58.062984 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.062957    6479 scope.go:122] "RemoveContainer" containerID="929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa"
Nov 01 20:49:58.063277 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:49:58.063261    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa\": container with ID starting with 929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa not found: ID does not exist" containerID="929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa"
Nov 01 20:49:58.063354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.063286    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa"} err="failed to get container status \"929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa\": rpc error: code = NotFound desc = could not find container \"929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa\": container with ID starting with 929b48ad38e12cea9a14cc55f523122357eb371f57e6c3683cfb43fdff6488aa not found: ID does not exist"
Nov 01 20:49:58.063354 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.063304    6479 scope.go:122] "RemoveContainer" containerID="5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"
Nov 01 20:49:58.063570 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:49:58.063547    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d\": container with ID starting with 5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d not found: ID does not exist" containerID="5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"
Nov 01 20:49:58.063617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.063577    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d"} err="failed to get container status \"5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d\": rpc error: code = NotFound desc = could not find container \"5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d\": container with ID starting with 5a92869873c2865d0d283699e077ca6cc3c374525351e4c46d05cd447d97f59d not found: ID does not exist"
Nov 01 20:49:58.110456 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.110427    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:58.141764 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.141735    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.141899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.141804    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.141899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.141849    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.141919    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.141981    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142024 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142005    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:49:58.142189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142036    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142090    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142131    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142189 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142172    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket\") pod \"96f33907-3c67-4187-923a-5f7c4e75b31e\" (UID: \"96f33907-3c67-4187-923a-5f7c4e75b31e\") "
Nov 01 20:49:58.142395 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142228    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:49:58.142561 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142536    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.142705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142689    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.142825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142588    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:49:58.142825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142604    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:49:58.143021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.142730    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:49:58.144059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.144037    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:49:58.144255 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.144236    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "kube-api-access-klzsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:49:58.144305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.144239    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:49:58.144305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.144269    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token" pod "96f33907-3c67-4187-923a-5f7c4e75b31e" (UID: "96f33907-3c67-4187-923a-5f7c4e75b31e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:49:58.244121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244084    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.244121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244118    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klzsn\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-kube-api-access-klzsn\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.244121 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244131    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.244367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244140    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.244367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244149    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f33907-3c67-4187-923a-5f7c4e75b31e-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.244367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244158    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f33907-3c67-4187-923a-5f7c4e75b31e-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.244367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.244167    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f33907-3c67-4187-923a-5f7c4e75b31e-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:58.348660 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.348618    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"]
Nov 01 20:49:58.352394 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.352359    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c268d1-openshift-default-76b6566dbc-qggsl"]
Nov 01 20:49:58.655770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.655688    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:58.721597 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:58.721187    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" path="/var/lib/kubelet/pods/96f33907-3c67-4187-923a-5f7c4e75b31e/volumes"
Nov 01 20:49:59.169566 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:49:59.169532    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:49:59.249699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:59.249638    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:49:59.951718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:49:59.951685    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:00.559506 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:00.559471    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:01.252313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:01.252280    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:01.862875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:01.862837    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:02.560045 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:02.560006    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:03.311487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:03.311456    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:03.851310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:03.851271    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:04.209527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:04.209492    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:04.665334 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:04.665255    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"]
Nov 01 20:50:04.665759 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:04.665552    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh" podUID="8406ce1e-cba2-404e-9a27-1fd30982f7ea" containerName="maas-api" containerID="cri-o://3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f" gracePeriod=30
Nov 01 20:50:04.708705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:04.708674    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:04.809219 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:04.809163    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:04.956779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:04.956457    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:50:05.009758 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.009726    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca\") pod \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") "
Nov 01 20:50:05.009971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.009786    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc\" (UniqueName: \"kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc\") pod \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") "
Nov 01 20:50:05.009971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.009846    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs\") pod \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") "
Nov 01 20:50:05.009971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.009879    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls\") pod \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\" (UID: \"8406ce1e-cba2-404e-9a27-1fd30982f7ea\") "
Nov 01 20:50:05.010176 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.010157    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca" pod "8406ce1e-cba2-404e-9a27-1fd30982f7ea" (UID: "8406ce1e-cba2-404e-9a27-1fd30982f7ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:05.012077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.012036    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs" pod "8406ce1e-cba2-404e-9a27-1fd30982f7ea" (UID: "8406ce1e-cba2-404e-9a27-1fd30982f7ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:50:05.012077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.012069    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc" pod "8406ce1e-cba2-404e-9a27-1fd30982f7ea" (UID: "8406ce1e-cba2-404e-9a27-1fd30982f7ea"). InnerVolumeSpecName "kube-api-access-tdjcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:05.012355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.012324    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls" pod "8406ce1e-cba2-404e-9a27-1fd30982f7ea" (UID: "8406ce1e-cba2-404e-9a27-1fd30982f7ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:50:05.059560 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.059527    6479 generic.go:356] "Generic (PLEG): container finished" podID="8406ce1e-cba2-404e-9a27-1fd30982f7ea" containerID="3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f" exitCode=0
Nov 01 20:50:05.059747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.059575    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"
Nov 01 20:50:05.059747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.059608    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh" event={"ID":"8406ce1e-cba2-404e-9a27-1fd30982f7ea","Type":"ContainerDied","Data":"3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f"}
Nov 01 20:50:05.059747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.059689    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh" event={"ID":"8406ce1e-cba2-404e-9a27-1fd30982f7ea","Type":"ContainerDied","Data":"8d98ffba3ec467c37a63291218120dca262693443d32c55bf7e846e9dd59bd7f"}
Nov 01 20:50:05.059747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.059715    6479 scope.go:122] "RemoveContainer" containerID="3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f"
Nov 01 20:50:05.068929 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.068908    6479 scope.go:122] "RemoveContainer" containerID="3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f"
Nov 01 20:50:05.069201 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:50:05.069181    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f\": container with ID starting with 3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f not found: ID does not exist" containerID="3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f"
Nov 01 20:50:05.069267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.069213    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f"} err="failed to get container status \"3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f\": rpc error: code = NotFound desc = could not find container \"3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f\": container with ID starting with 3f3760d1b2ea5d4ab3ee89fed72698b437cba68c565cf29897206defc6bfcb3f not found: ID does not exist"
Nov 01 20:50:05.081714 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.081685    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"]
Nov 01 20:50:05.090737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.087862    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a97e34-b59d9c7f-6z4vh"]
Nov 01 20:50:05.110599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.110573    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8406ce1e-cba2-404e-9a27-1fd30982f7ea-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.110599 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.110598    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdjcc\" (UniqueName: \"kubernetes.io/projected/8406ce1e-cba2-404e-9a27-1fd30982f7ea-kube-api-access-tdjcc\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.110761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.110610    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.110761 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:05.110619    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8406ce1e-cba2-404e-9a27-1fd30982f7ea-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:06.043048 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:06.043013    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:50:06.705846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:06.705810    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8406ce1e-cba2-404e-9a27-1fd30982f7ea" path="/var/lib/kubelet/pods/8406ce1e-cba2-404e-9a27-1fd30982f7ea/volumes"
Nov 01 20:50:09.321031 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:09.320992    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:09.397062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:09.397012    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:09.724124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:09.724093    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:10.048754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:10.048666    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"]
Nov 01 20:50:10.048957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:10.048929    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" containerID="cri-o://929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017" gracePeriod=30
Nov 01 20:50:10.084943 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:10.084894    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:11.543424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543387    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-978cf677d-rwtqn"]
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543778    6479 cpu_manager.go:414] "containerMap: removing container" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543789    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543799    6479 cpu_manager.go:414] "containerMap: removing container" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543804    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543819    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f96002ef-603e-4812-99b1-21ca3009ee8e" containerName="maas-api"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543824    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96002ef-603e-4812-99b1-21ca3009ee8e" containerName="maas-api"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543836    6479 cpu_manager.go:414] "containerMap: removing container" podUID="8406ce1e-cba2-404e-9a27-1fd30982f7ea" containerName="maas-api"
Nov 01 20:50:11.543853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543841    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8406ce1e-cba2-404e-9a27-1fd30982f7ea" containerName="maas-api"
Nov 01 20:50:11.544158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543901    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96002ef-603e-4812-99b1-21ca3009ee8e" containerName="maas-api"
Nov 01 20:50:11.544158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543911    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="8406ce1e-cba2-404e-9a27-1fd30982f7ea" containerName="maas-api"
Nov 01 20:50:11.544158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543920    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy"
Nov 01 20:50:11.544158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.543927    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f33907-3c67-4187-923a-5f7c4e75b31e" containerName="istio-proxy"
Nov 01 20:50:11.548263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.548246    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.554342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.554230    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-978cf677d-rwtqn"]
Nov 01 20:50:11.555948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.555923    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vzlvj\""
Nov 01 20:50:11.555948 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.555934    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 20:50:11.584665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.584614    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgv4h\" (UniqueName: \"kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h\") pod \"payload-pre-processing-978cf677d-rwtqn\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") " pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.584810 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.584685    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume\") pod \"payload-pre-processing-978cf677d-rwtqn\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") " pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.685742 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.685712    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv4h\" (UniqueName: \"kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h\") pod \"payload-pre-processing-978cf677d-rwtqn\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") " pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.685910 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.685756    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume\") pod \"payload-pre-processing-978cf677d-rwtqn\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") " pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.686427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.686408    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume\") pod \"payload-pre-processing-978cf677d-rwtqn\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") " pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.693196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.693165    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv4h\" (UniqueName: \"kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h\") pod \"payload-pre-processing-978cf677d-rwtqn\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") " pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:11.867710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:11.867621    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:12.140249 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:12.140153    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-978cf677d-rwtqn"]
Nov 01 20:50:12.144138 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:50:12.144113    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf96c8b6_f7be_45fb_90f0_8518f3930055.slice/crio-5a524cf864848a458cc74c81553e3acbdd0f3dcbbc29ab0d7829dd67c0c28af8 WatchSource:0}: Error finding container 5a524cf864848a458cc74c81553e3acbdd0f3dcbbc29ab0d7829dd67c0c28af8: Status 404 returned error can't find the container with id 5a524cf864848a458cc74c81553e3acbdd0f3dcbbc29ab0d7829dd67c0c28af8
Nov 01 20:50:13.102596 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:13.102562    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn" event={"ID":"df96c8b6-f7be-45fb-90f0-8518f3930055","Type":"ContainerStarted","Data":"b79753cf2e9ae7d584bc8bd38243c6b741f7d98f8c8c958634ef435abaeba8cf"}
Nov 01 20:50:13.103000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:13.102610    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn" event={"ID":"df96c8b6-f7be-45fb-90f0-8518f3930055","Type":"ContainerStarted","Data":"5a524cf864848a458cc74c81553e3acbdd0f3dcbbc29ab0d7829dd67c0c28af8"}
Nov 01 20:50:13.103000 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:13.102678    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:13.117957 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:13.117909    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn" podStartSLOduration=2.117896555 podStartE2EDuration="2.117896555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:13.11721795 +0000 UTC m=+3058.532850425" watchObservedRunningTime="2026-11-01 20:50:13.117896555 +0000 UTC m=+3058.533529062"
Nov 01 20:50:14.659718 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:14.659681    6479 patch_prober.go:29] interesting pod/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:50:14.660080 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:14.659735    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:50:15.299229 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.299201    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:50:15.422777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422681    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.422777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422761    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422797    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422832    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422866    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422923    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.422942    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:50:15.423022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423010    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423043    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423097    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:50:15.423370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423186    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:50:15.423370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423201    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:50:15.423370 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423210    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:15.423634 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423433    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc\") pod \"df2a4073-81ad-47b0-86c7-0c7154748c5b\" (UID: \"df2a4073-81ad-47b0-86c7-0c7154748c5b\") "
Nov 01 20:50:15.423824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423796    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2a4073-81ad-47b0-86c7-0c7154748c5b-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.423824 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423818    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.423930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423832    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.423930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423848    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.423930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.423862    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.425260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.425232    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:50:15.425260 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.425235    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:15.425510 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.425488    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:50:15.425582 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.425509    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc" pod "df2a4073-81ad-47b0-86c7-0c7154748c5b" (UID: "df2a4073-81ad-47b0-86c7-0c7154748c5b"). InnerVolumeSpecName "kube-api-access-rgrwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:15.525013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.524977    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.525013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.525006    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.525013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.525017    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrwc\" (UniqueName: \"kubernetes.io/projected/df2a4073-81ad-47b0-86c7-0c7154748c5b-kube-api-access-rgrwc\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:15.525235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:15.525027    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2a4073-81ad-47b0-86c7-0c7154748c5b-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:16.117443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.117402    6479 generic.go:356] "Generic (PLEG): container finished" podID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerID="929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017" exitCode=0
Nov 01 20:50:16.117853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.117458    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"
Nov 01 20:50:16.117853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.117483    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" event={"ID":"df2a4073-81ad-47b0-86c7-0c7154748c5b","Type":"ContainerDied","Data":"929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017"}
Nov 01 20:50:16.117853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.117541    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs" event={"ID":"df2a4073-81ad-47b0-86c7-0c7154748c5b","Type":"ContainerDied","Data":"efd1507fd78429e7e8cdb54075df42ea0ceab5aea6e146cb5bba3a58b66d340c"}
Nov 01 20:50:16.117853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.117563    6479 scope.go:122] "RemoveContainer" containerID="929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017"
Nov 01 20:50:16.126566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.126548    6479 scope.go:122] "RemoveContainer" containerID="751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"
Nov 01 20:50:16.134443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.134427    6479 scope.go:122] "RemoveContainer" containerID="929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017"
Nov 01 20:50:16.134713 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:50:16.134693    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017\": container with ID starting with 929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017 not found: ID does not exist" containerID="929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017"
Nov 01 20:50:16.134753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.134725    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017"} err="failed to get container status \"929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017\": rpc error: code = NotFound desc = could not find container \"929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017\": container with ID starting with 929306bdf0d686c73298d5ef2a85b7b705a06aca152a112316fc4460e8085017 not found: ID does not exist"
Nov 01 20:50:16.134753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.134743    6479 scope.go:122] "RemoveContainer" containerID="751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"
Nov 01 20:50:16.134987 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:50:16.134972    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96\": container with ID starting with 751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96 not found: ID does not exist" containerID="751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"
Nov 01 20:50:16.135027 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.134995    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96"} err="failed to get container status \"751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96\": rpc error: code = NotFound desc = could not find container \"751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96\": container with ID starting with 751b1570c0f3aaa9292fd495cf7693ddb674e9df9b7e6fc58908f7c24bf7df96 not found: ID does not exist"
Nov 01 20:50:16.139817 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.139796    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"]
Nov 01 20:50:16.143074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.143050    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a97e34-openshift-default-949f9f895-r72hs"]
Nov 01 20:50:16.706711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:16.706677    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" path="/var/lib/kubelet/pods/df2a4073-81ad-47b0-86c7-0c7154748c5b/volumes"
Nov 01 20:50:23.986368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:23.986321    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:24.110447 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:24.110417    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:25.788729 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.788626    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-68b4f896d-pxw6w"]
Nov 01 20:50:25.789245 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.789223    6479 cpu_manager.go:414] "containerMap: removing container" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy"
Nov 01 20:50:25.789328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.789248    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy"
Nov 01 20:50:25.789328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.789276    6479 cpu_manager.go:414] "containerMap: removing container" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy"
Nov 01 20:50:25.789328 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.789285    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy"
Nov 01 20:50:25.789498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.789375    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy"
Nov 01 20:50:25.789498 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.789393    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2a4073-81ad-47b0-86c7-0c7154748c5b" containerName="istio-proxy"
Nov 01 20:50:25.793710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.793687    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:25.799418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.799393    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68b4f896d-pxw6w"]
Nov 01 20:50:25.927531 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.927493    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume\") pod \"payload-processing-68b4f896d-pxw6w\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") " pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:25.927762 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:25.927628    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ttf\" (UniqueName: \"kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf\") pod \"payload-processing-68b4f896d-pxw6w\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") " pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:26.030107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:26.028803    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ttf\" (UniqueName: \"kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf\") pod \"payload-processing-68b4f896d-pxw6w\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") " pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:26.030107 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:26.028952    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume\") pod \"payload-processing-68b4f896d-pxw6w\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") " pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:26.035179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:26.030381    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume\") pod \"payload-processing-68b4f896d-pxw6w\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") " pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:26.037526 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:26.037484    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ttf\" (UniqueName: \"kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf\") pod \"payload-processing-68b4f896d-pxw6w\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") " pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:26.109565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:26.109482    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:26.379211 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:26.379133    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68b4f896d-pxw6w"]
Nov 01 20:50:26.382327 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:50:26.382298    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a718ee_c308_46b4_9a46_af14ea42d509.slice/crio-1a9e051823c945b0699d870283c6754efd7d81f42b4877de87420737a09d8018 WatchSource:0}: Error finding container 1a9e051823c945b0699d870283c6754efd7d81f42b4877de87420737a09d8018: Status 404 returned error can't find the container with id 1a9e051823c945b0699d870283c6754efd7d81f42b4877de87420737a09d8018
Nov 01 20:50:27.173275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:27.173237    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w" event={"ID":"f4a718ee-c308-46b4-9a46-af14ea42d509","Type":"ContainerStarted","Data":"8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1"}
Nov 01 20:50:27.173789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:27.173298    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w" event={"ID":"f4a718ee-c308-46b4-9a46-af14ea42d509","Type":"ContainerStarted","Data":"1a9e051823c945b0699d870283c6754efd7d81f42b4877de87420737a09d8018"}
Nov 01 20:50:27.173789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:27.173359    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:27.190433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:27.190384    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w" podStartSLOduration=2.190373182 podStartE2EDuration="2.190373182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:27.188949702 +0000 UTC m=+3072.604582190" watchObservedRunningTime="2026-11-01 20:50:27.190373182 +0000 UTC m=+3072.606005655"
Nov 01 20:50:29.170300 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:50:29.170261    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:50:35.957788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:35.957741    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:38.180437 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:38.180401    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:39.999344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:39.999304    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"]
Nov 01 20:50:40.003171 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.003145    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.011321 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.011294    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"]
Nov 01 20:50:40.072043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.072008    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c77aedf-651e-490c-a0e7-31afeacf0102-plugins-config-volume\") pod \"payload-processing-5ff9ddc8d7-c89gd\" (UID: \"7c77aedf-651e-490c-a0e7-31afeacf0102\") " pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.072043 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.072044    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j99v\" (UniqueName: \"kubernetes.io/projected/7c77aedf-651e-490c-a0e7-31afeacf0102-kube-api-access-7j99v\") pod \"payload-processing-5ff9ddc8d7-c89gd\" (UID: \"7c77aedf-651e-490c-a0e7-31afeacf0102\") " pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.155006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.154968    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7f75799d9b-w7752"]
Nov 01 20:50:40.158521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.158495    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.165304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.165258    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7f75799d9b-w7752"]
Nov 01 20:50:40.173103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.173072    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72958\" (UniqueName: \"kubernetes.io/projected/d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b-kube-api-access-72958\") pod \"payload-pre-processing-7f75799d9b-w7752\" (UID: \"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b\") " pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.173256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.173234    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b-plugins-config-volume\") pod \"payload-pre-processing-7f75799d9b-w7752\" (UID: \"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b\") " pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.173433 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.173410    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c77aedf-651e-490c-a0e7-31afeacf0102-plugins-config-volume\") pod \"payload-processing-5ff9ddc8d7-c89gd\" (UID: \"7c77aedf-651e-490c-a0e7-31afeacf0102\") " pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.173557 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.173467    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j99v\" (UniqueName: \"kubernetes.io/projected/7c77aedf-651e-490c-a0e7-31afeacf0102-kube-api-access-7j99v\") pod \"payload-processing-5ff9ddc8d7-c89gd\" (UID: \"7c77aedf-651e-490c-a0e7-31afeacf0102\") " pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.175112 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.175088    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c77aedf-651e-490c-a0e7-31afeacf0102-plugins-config-volume\") pod \"payload-processing-5ff9ddc8d7-c89gd\" (UID: \"7c77aedf-651e-490c-a0e7-31afeacf0102\") " pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.181832 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.181809    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j99v\" (UniqueName: \"kubernetes.io/projected/7c77aedf-651e-490c-a0e7-31afeacf0102-kube-api-access-7j99v\") pod \"payload-processing-5ff9ddc8d7-c89gd\" (UID: \"7c77aedf-651e-490c-a0e7-31afeacf0102\") " pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.274420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.274338    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72958\" (UniqueName: \"kubernetes.io/projected/d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b-kube-api-access-72958\") pod \"payload-pre-processing-7f75799d9b-w7752\" (UID: \"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b\") " pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.274420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.274396    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b-plugins-config-volume\") pod \"payload-pre-processing-7f75799d9b-w7752\" (UID: \"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b\") " pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.275168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.275146    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b-plugins-config-volume\") pod \"payload-pre-processing-7f75799d9b-w7752\" (UID: \"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b\") " pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.282195 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.282168    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72958\" (UniqueName: \"kubernetes.io/projected/d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b-kube-api-access-72958\") pod \"payload-pre-processing-7f75799d9b-w7752\" (UID: \"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b\") " pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.317074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.317042    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:40.472839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.472799    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:40.587944 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.587920    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"]
Nov 01 20:50:40.590056 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:50:40.590027    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c77aedf_651e_490c_a0e7_31afeacf0102.slice/crio-79015536cd228ac5e8730265c64e06908a401813df7797d6ce69b0a589eda8fb WatchSource:0}: Error finding container 79015536cd228ac5e8730265c64e06908a401813df7797d6ce69b0a589eda8fb: Status 404 returned error can't find the container with id 79015536cd228ac5e8730265c64e06908a401813df7797d6ce69b0a589eda8fb
Nov 01 20:50:40.742446 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:40.742415    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7f75799d9b-w7752"]
Nov 01 20:50:40.745573 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:50:40.745544    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80d2eeb_9b5f_4d37_85bf_c6eb910cdd5b.slice/crio-9991f2b3f7ba341f03ab34b4e8363a97af54312bde95d814cbdea107ea3d5aa6 WatchSource:0}: Error finding container 9991f2b3f7ba341f03ab34b4e8363a97af54312bde95d814cbdea107ea3d5aa6: Status 404 returned error can't find the container with id 9991f2b3f7ba341f03ab34b4e8363a97af54312bde95d814cbdea107ea3d5aa6
Nov 01 20:50:41.238901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.238857    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd" event={"ID":"7c77aedf-651e-490c-a0e7-31afeacf0102","Type":"ContainerStarted","Data":"7013a06e2f94da5105a07f6398d8de2ff2518a2cb25dec604d04957807ff4aae"}
Nov 01 20:50:41.239355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.238926    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd" event={"ID":"7c77aedf-651e-490c-a0e7-31afeacf0102","Type":"ContainerStarted","Data":"79015536cd228ac5e8730265c64e06908a401813df7797d6ce69b0a589eda8fb"}
Nov 01 20:50:41.239355 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.238955    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:41.240311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.240286    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752" event={"ID":"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b","Type":"ContainerStarted","Data":"93d083a4dc16e39c1b49ce18b9e03f4f21e73b3add9789b2f80b18536dcc261c"}
Nov 01 20:50:41.240424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.240326    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752" event={"ID":"d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b","Type":"ContainerStarted","Data":"9991f2b3f7ba341f03ab34b4e8363a97af54312bde95d814cbdea107ea3d5aa6"}
Nov 01 20:50:41.240424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.240356    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:41.255855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.255805    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd" podStartSLOduration=2.25579154 podStartE2EDuration="2.25579154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:41.253969086 +0000 UTC m=+3086.669601560" watchObservedRunningTime="2026-11-01 20:50:41.25579154 +0000 UTC m=+3086.671424012"
Nov 01 20:50:41.269239 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:41.269184    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752" podStartSLOduration=1.269167172 podStartE2EDuration="1.269167172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:41.268234872 +0000 UTC m=+3086.683867345" watchObservedRunningTime="2026-11-01 20:50:41.269167172 +0000 UTC m=+3086.684799649"
Nov 01 20:50:52.248389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:52.248348    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5ff9ddc8d7-c89gd"
Nov 01 20:50:52.248809 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:52.248524    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7f75799d9b-w7752"
Nov 01 20:50:52.294705 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:52.294662    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-68b4f896d-pxw6w"]
Nov 01 20:50:52.306226 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:52.306189    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-978cf677d-rwtqn"]
Nov 01 20:50:54.361855 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.361814    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:54.528879 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.528844    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:54.703330 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.703286    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:50:54.749310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.749275    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"]
Nov 01 20:50:54.752861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.752839    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.757042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.757015    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 20:50:54.759778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.759756    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 20:50:54.760022 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.759764    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 20:50:54.762202 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.762181    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-59hhh\""
Nov 01 20:50:54.762309 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.762258    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 01 20:50:54.764068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.764048    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"]
Nov 01 20:50:54.826638 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.826552    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.826873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.826706    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.826873 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.826761    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hctg\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.827006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.826947    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.857283 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.857249    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 20:50:54.928443 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.928393    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.928669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.928448    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.928669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.928473    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hctg\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.928669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.928541    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.929311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.929284    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.929407 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.929362    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.931545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.931518    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:54.939914 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:54.939890    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hctg\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg\") pod \"payload-processing-external-model-5bcd7997-9gj26\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:55.070134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:55.070043    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:55.346495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:55.346463    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"]
Nov 01 20:50:55.348742 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:50:55.348711    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903d17c3_fb78_4b65_b1c4_9508822af912.slice/crio-5e076802917c42b0cadf2c2f34680c8540056daf4a0841c707640548e54b8743 WatchSource:0}: Error finding container 5e076802917c42b0cadf2c2f34680c8540056daf4a0841c707640548e54b8743: Status 404 returned error can't find the container with id 5e076802917c42b0cadf2c2f34680c8540056daf4a0841c707640548e54b8743
Nov 01 20:50:55.937051 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:55.937005    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:50:56.328564 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:56.328517    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26" event={"ID":"903d17c3-fb78-4b65-b1c4-9508822af912","Type":"ContainerStarted","Data":"af47b4abd2b235a4a84320fb11ddaa500a7e2e9fbd290e12b282c38521190026"}
Nov 01 20:50:56.328791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:56.328627    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26" event={"ID":"903d17c3-fb78-4b65-b1c4-9508822af912","Type":"ContainerStarted","Data":"5e076802917c42b0cadf2c2f34680c8540056daf4a0841c707640548e54b8743"}
Nov 01 20:50:56.328882 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:56.328861    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:50:56.347950 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:56.347879    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26" podStartSLOduration=2.347861498 podStartE2EDuration="2.347861498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:56.346084012 +0000 UTC m=+3101.761716483" watchObservedRunningTime="2026-11-01 20:50:56.347861498 +0000 UTC m=+3101.763493972"
Nov 01 20:50:57.297782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:57.297710    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w" podUID="f4a718ee-c308-46b4-9a46-af14ea42d509" containerName="payload-processing" containerID="cri-o://8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1" gracePeriod=25
Nov 01 20:50:57.319439 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:57.319188    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn" podUID="df96c8b6-f7be-45fb-90f0-8518f3930055" containerName="payload-pre-processing" containerID="cri-o://b79753cf2e9ae7d584bc8bd38243c6b741f7d98f8c8c958634ef435abaeba8cf" gracePeriod=25
Nov 01 20:50:58.112685 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.098964    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:58.153672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.153430    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume\") pod \"f4a718ee-c308-46b4-9a46-af14ea42d509\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") "
Nov 01 20:50:58.153672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.153525    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf\" (UniqueName: \"kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf\") pod \"f4a718ee-c308-46b4-9a46-af14ea42d509\" (UID: \"f4a718ee-c308-46b4-9a46-af14ea42d509\") "
Nov 01 20:50:58.166089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.166039    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume" pod "f4a718ee-c308-46b4-9a46-af14ea42d509" (UID: "f4a718ee-c308-46b4-9a46-af14ea42d509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:58.166848 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.166826    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a718ee-c308-46b4-9a46-af14ea42d509-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:58.235874 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.235816    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf" pod "f4a718ee-c308-46b4-9a46-af14ea42d509" (UID: "f4a718ee-c308-46b4-9a46-af14ea42d509"). InnerVolumeSpecName "kube-api-access-68ttf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:58.290716 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.277951    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ttf\" (UniqueName: \"kubernetes.io/projected/f4a718ee-c308-46b4-9a46-af14ea42d509-kube-api-access-68ttf\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:58.464552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.463805    6479 generic.go:356] "Generic (PLEG): container finished" podID="f4a718ee-c308-46b4-9a46-af14ea42d509" containerID="8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1" exitCode=0
Nov 01 20:50:58.464552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.463922    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w" event={"ID":"f4a718ee-c308-46b4-9a46-af14ea42d509","Type":"ContainerDied","Data":"8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1"}
Nov 01 20:50:58.464552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.463942    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w"
Nov 01 20:50:58.464552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.463977    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68b4f896d-pxw6w" event={"ID":"f4a718ee-c308-46b4-9a46-af14ea42d509","Type":"ContainerDied","Data":"1a9e051823c945b0699d870283c6754efd7d81f42b4877de87420737a09d8018"}
Nov 01 20:50:58.464552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.464010    6479 scope.go:122] "RemoveContainer" containerID="8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1"
Nov 01 20:50:58.556678 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.554808    6479 scope.go:122] "RemoveContainer" containerID="8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1"
Nov 01 20:50:58.562683 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:50:58.562227    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1\": container with ID starting with 8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1 not found: ID does not exist" containerID="8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1"
Nov 01 20:50:58.562683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.562286    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1"} err="failed to get container status \"8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1\": rpc error: code = NotFound desc = could not find container \"8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1\": container with ID starting with 8a26ffbbce693a516ff2459922c68aa7895448d52db351b6e3ccd2eab0f95bf1 not found: ID does not exist"
Nov 01 20:50:58.628695 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.628277    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-68b4f896d-pxw6w"]
Nov 01 20:50:58.633672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.631019    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-68b4f896d-pxw6w"]
Nov 01 20:50:58.726671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:58.725801    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a718ee-c308-46b4-9a46-af14ea42d509" path="/var/lib/kubelet/pods/f4a718ee-c308-46b4-9a46-af14ea42d509/volumes"
Nov 01 20:50:59.453023 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:50:59.450192    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:50:59.501173 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.501136    6479 generic.go:356] "Generic (PLEG): container finished" podID="df96c8b6-f7be-45fb-90f0-8518f3930055" containerID="b79753cf2e9ae7d584bc8bd38243c6b741f7d98f8c8c958634ef435abaeba8cf" exitCode=0
Nov 01 20:50:59.517672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.517605    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn" event={"ID":"df96c8b6-f7be-45fb-90f0-8518f3930055","Type":"ContainerDied","Data":"b79753cf2e9ae7d584bc8bd38243c6b741f7d98f8c8c958634ef435abaeba8cf"}
Nov 01 20:50:59.616711 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.616685    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:50:59.653214 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.653130    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume\") pod \"df96c8b6-f7be-45fb-90f0-8518f3930055\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") "
Nov 01 20:50:59.653613 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.653590    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume" pod "df96c8b6-f7be-45fb-90f0-8518f3930055" (UID: "df96c8b6-f7be-45fb-90f0-8518f3930055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:59.753960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.753926    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h\" (UniqueName: \"kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h\") pod \"df96c8b6-f7be-45fb-90f0-8518f3930055\" (UID: \"df96c8b6-f7be-45fb-90f0-8518f3930055\") "
Nov 01 20:50:59.754305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.754284    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df96c8b6-f7be-45fb-90f0-8518f3930055-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:59.756113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.756085    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h" pod "df96c8b6-f7be-45fb-90f0-8518f3930055" (UID: "df96c8b6-f7be-45fb-90f0-8518f3930055"). InnerVolumeSpecName "kube-api-access-tgv4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:59.855068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:50:59.855033    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgv4h\" (UniqueName: \"kubernetes.io/projected/df96c8b6-f7be-45fb-90f0-8518f3930055-kube-api-access-tgv4h\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:51:00.529052 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:00.529015    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn" event={"ID":"df96c8b6-f7be-45fb-90f0-8518f3930055","Type":"ContainerDied","Data":"5a524cf864848a458cc74c81553e3acbdd0f3dcbbc29ab0d7829dd67c0c28af8"}
Nov 01 20:51:00.529521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:00.529056    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-978cf677d-rwtqn"
Nov 01 20:51:00.529521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:00.529078    6479 scope.go:122] "RemoveContainer" containerID="b79753cf2e9ae7d584bc8bd38243c6b741f7d98f8c8c958634ef435abaeba8cf"
Nov 01 20:51:00.550476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:00.550446    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-978cf677d-rwtqn"]
Nov 01 20:51:00.553618 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:00.553594    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-978cf677d-rwtqn"]
Nov 01 20:51:00.705892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:00.705858    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df96c8b6-f7be-45fb-90f0-8518f3930055" path="/var/lib/kubelet/pods/df96c8b6-f7be-45fb-90f0-8518f3930055/volumes"
Nov 01 20:51:07.364969 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:07.364940    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:51:25.378863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:51:25.378816    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:51:29.169298 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:51:29.169254    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:51:59.169263 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:51:59.169183    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:52:24.098124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:24.098088    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:52:28.327329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:28.326953    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"]
Nov 01 20:52:28.350298 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:28.350257    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:52:28.451137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:28.451104    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:52:28.632389 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:28.632301    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:52:28.655503 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:28.655470    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:52:29.185337 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:52:29.185292    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:52:33.328671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:33.327977    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26" podUID="903d17c3-fb78-4b65-b1c4-9508822af912" containerName="payload-processing" containerID="cri-o://af47b4abd2b235a4a84320fb11ddaa500a7e2e9fbd290e12b282c38521190026" gracePeriod=25
Nov 01 20:52:34.150524 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.150439    6479 generic.go:356] "Generic (PLEG): container finished" podID="903d17c3-fb78-4b65-b1c4-9508822af912" containerID="af47b4abd2b235a4a84320fb11ddaa500a7e2e9fbd290e12b282c38521190026" exitCode=0
Nov 01 20:52:34.150838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.150809    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26" event={"ID":"903d17c3-fb78-4b65-b1c4-9508822af912","Type":"ContainerDied","Data":"af47b4abd2b235a4a84320fb11ddaa500a7e2e9fbd290e12b282c38521190026"}
Nov 01 20:52:34.372108 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.372077    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:52:34.533670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.527956    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume" pod "903d17c3-fb78-4b65-b1c4-9508822af912" (UID: "903d17c3-fb78-4b65-b1c4-9508822af912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:52:34.533670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.528034    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume\") pod \"903d17c3-fb78-4b65-b1c4-9508822af912\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") "
Nov 01 20:52:34.533670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.528158    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg\") pod \"903d17c3-fb78-4b65-b1c4-9508822af912\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") "
Nov 01 20:52:34.533670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.528220    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials\") pod \"903d17c3-fb78-4b65-b1c4-9508822af912\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") "
Nov 01 20:52:34.533670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.528422    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume\") pod \"903d17c3-fb78-4b65-b1c4-9508822af912\" (UID: \"903d17c3-fb78-4b65-b1c4-9508822af912\") "
Nov 01 20:52:34.534149 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.533860    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:34.547712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.547681    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume" pod "903d17c3-fb78-4b65-b1c4-9508822af912" (UID: "903d17c3-fb78-4b65-b1c4-9508822af912"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:52:34.573495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.573440    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg" pod "903d17c3-fb78-4b65-b1c4-9508822af912" (UID: "903d17c3-fb78-4b65-b1c4-9508822af912"). InnerVolumeSpecName "kube-api-access-9hctg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:34.573820 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.573781    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials" pod "903d17c3-fb78-4b65-b1c4-9508822af912" (UID: "903d17c3-fb78-4b65-b1c4-9508822af912"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:34.648669 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.637250    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hctg\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-kube-api-access-9hctg\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:34.754691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.743407    6479 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d17c3-fb78-4b65-b1c4-9508822af912-provider-credentials\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:34.754691 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:34.743444    6479 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d17c3-fb78-4b65-b1c4-9508822af912-routing-overlay-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:35.166006 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:35.165909    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26" event={"ID":"903d17c3-fb78-4b65-b1c4-9508822af912","Type":"ContainerDied","Data":"5e076802917c42b0cadf2c2f34680c8540056daf4a0841c707640548e54b8743"}
Nov 01 20:52:35.166281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:35.166261    6479 scope.go:122] "RemoveContainer" containerID="af47b4abd2b235a4a84320fb11ddaa500a7e2e9fbd290e12b282c38521190026"
Nov 01 20:52:35.166566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:35.166546    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"
Nov 01 20:52:35.212021 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:35.210865    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"]
Nov 01 20:52:35.223668 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:35.221902    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-9gj26"]
Nov 01 20:52:35.359047 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:35.358988    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:52:36.706227 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:36.706191    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903d17c3-fb78-4b65-b1c4-9508822af912" path="/var/lib/kubelet/pods/903d17c3-fb78-4b65-b1c4-9508822af912/volumes"
Nov 01 20:52:48.421178 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:48.421140    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:52:50.533175 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:50.533033    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:52:50.602057 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:50.601946    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:52:54.157074 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:52:54.157029    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:52:59.169444 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:52:59.169405    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:53:00.203972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:00.203939    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:03.159897 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:03.159843    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-s4pct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:53:03.160546 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:03.160510    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-s4pct" podUID="020b4fc0-677d-4ec2-ac2d-bdd1379e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:53:03.161194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:03.161160    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jsxtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:53:03.161374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:03.161348    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jsxtq" podUID="838f2059-c8d7-4b2c-95e9-a9a31c87683c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:53:21.510900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:21.510864    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:24.262712 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:24.262672    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:53:29.168574 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:53:29.168489    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:53:31.727412 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:31.727355    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:35.492670 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:35.491701    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:35.792885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:35.792774    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:53:35.793190 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:35.793139    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:53:52.132572 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:52.132532    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:55.832674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:55.828781    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:58.931282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:53:58.931233    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:53:59.169574 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:53:59.169535    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:54:09.749326 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:09.749285    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:54:23.434186 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:23.434148    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:54:24.343565 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:24.343522    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:54:26.972062 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:26.972020    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:54:28.356960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:28.356919    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:54:28.422942 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:28.422906    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:54:29.175443 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:54:29.175405    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:54:30.337579 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:30.337542    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:54:48.602523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:48.602476    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:54:52.735440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:54:52.735394    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:54:59.169217 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:54:59.169137    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:55:05.107132 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:05.107095    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:08.128153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:08.128115    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:11.920960 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:11.920923    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:24.272275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:24.272223    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:26.924338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:26.924299    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:29.278188 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:55:29.278148    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:55:33.145363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:33.145332    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:37.137424 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:37.137389    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:44.425552 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:44.425506    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:44.497604 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:44.497568    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:52.207487 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:52.207447    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:56.127418 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.127366    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:55:56.253815 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.253777    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:56.612578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.612530    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"]
Nov 01 20:55:56.613192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.613164    6479 cpu_manager.go:414] "containerMap: removing container" podUID="df96c8b6-f7be-45fb-90f0-8518f3930055" containerName="payload-pre-processing"
Nov 01 20:55:56.613419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.613398    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df96c8b6-f7be-45fb-90f0-8518f3930055" containerName="payload-pre-processing"
Nov 01 20:55:56.613913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.613886    6479 cpu_manager.go:414] "containerMap: removing container" podUID="903d17c3-fb78-4b65-b1c4-9508822af912" containerName="payload-processing"
Nov 01 20:55:56.613913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.613912    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903d17c3-fb78-4b65-b1c4-9508822af912" containerName="payload-processing"
Nov 01 20:55:56.614072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.613947    6479 cpu_manager.go:414] "containerMap: removing container" podUID="f4a718ee-c308-46b4-9a46-af14ea42d509" containerName="payload-processing"
Nov 01 20:55:56.614072 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.613955    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a718ee-c308-46b4-9a46-af14ea42d509" containerName="payload-processing"
Nov 01 20:55:56.614158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.614077    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="df96c8b6-f7be-45fb-90f0-8518f3930055" containerName="payload-pre-processing"
Nov 01 20:55:56.614158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.614093    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="903d17c3-fb78-4b65-b1c4-9508822af912" containerName="payload-processing"
Nov 01 20:55:56.614158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.614107    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a718ee-c308-46b4-9a46-af14ea42d509" containerName="payload-processing"
Nov 01 20:55:56.627042 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.627022    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.629544 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.629515    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"]
Nov 01 20:55:56.634460 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.634436    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4e9e72-openshift-default-dockercfg-gp4wz\""
Nov 01 20:55:56.656554 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.656523    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:56.667778 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.667752    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.667885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.667866    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.667996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.667918    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9vt\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.667996 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.667947    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.668119 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.668071    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.668179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.668139    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.668243 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.668226    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.668292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.668248    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.668292 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.668277    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.743640 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.743606    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:56.768919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.768892    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.768946    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.768965    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9vt\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769093 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.768983    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769267 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769150    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769329 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769262    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769335    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769359    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769401 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769366    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769580 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769445    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769619    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769797 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769771    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.769965 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.769946    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.770103 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.770080    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.771365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.771343    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.771577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.771560    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.776735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.776711    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.776819 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.776784    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9vt\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt\") pod \"e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") " pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:56.948305 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:56.948274    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:57.224770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.224625    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"]
Nov 01 20:55:57.227535 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:55:57.227507    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c95c85_ec0e_4250_9f99_2fda96b2c327.slice/crio-1ef3b47e6557e87d122a829c8d1a65f5caa68c1b3508e76892e1ed820902d005 WatchSource:0}: Error finding container 1ef3b47e6557e87d122a829c8d1a65f5caa68c1b3508e76892e1ed820902d005: Status 404 returned error can't find the container with id 1ef3b47e6557e87d122a829c8d1a65f5caa68c1b3508e76892e1ed820902d005
Nov 01 20:55:57.229823 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.229789    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:55:57.229917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.229857    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:55:57.229917 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.229885    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:55:57.488674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.488554    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" event={"ID":"79c95c85-ec0e-4250-9f99-2fda96b2c327","Type":"ContainerStarted","Data":"3dd95819de66cde32d437d5c2249369e9ee68342b9e8199d1f97b951025f952f"}
Nov 01 20:55:57.488674 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.488623    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" event={"ID":"79c95c85-ec0e-4250-9f99-2fda96b2c327","Type":"ContainerStarted","Data":"1ef3b47e6557e87d122a829c8d1a65f5caa68c1b3508e76892e1ed820902d005"}
Nov 01 20:55:57.506376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.506316    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podStartSLOduration=1.506300647 podStartE2EDuration="1.506300647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:55:57.504784281 +0000 UTC m=+3402.920416755" watchObservedRunningTime="2026-11-01 20:55:57.506300647 +0000 UTC m=+3402.921933114"
Nov 01 20:55:57.949094 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:57.949063    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:58.953657 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:58.953623    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:59.173254 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:55:59.173217    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:55:59.509187 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:59.509148    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:59.510302 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:59.510277    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:55:59.644748 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:59.644713    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:55:59.745574 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:55:59.745542    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:00.900527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.900487    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"]
Nov 01 20:56:00.904533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.904501    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:00.913934 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.913909    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4e9e72-serving-cert\""
Nov 01 20:56:00.915440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.915412    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:00.917445 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.917409    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"]
Nov 01 20:56:00.951013 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.950978    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"]
Nov 01 20:56:00.955340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.955314    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:00.964402 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.962868    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e9e72\""
Nov 01 20:56:00.964853 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:00.964823    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"]
Nov 01 20:56:01.009463 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.009425    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:01.011687 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.011624    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.011847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.011791    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.011906 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.011847    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk5xq\" (UniqueName: \"kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.011985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.011968    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.012030 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.012005    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.012100 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.012080    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxr2\" (UniqueName: \"kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.013838 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.013818    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bbf535b9-dockercfg-vr8x9\""
Nov 01 20:56:01.109847 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.109814    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:01.113078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113055    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.113143 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113098    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk5xq\" (UniqueName: \"kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.113192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113177    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.113235 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113218    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.113290 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113253    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxr2\" (UniqueName: \"kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.113420 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113395    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.114014 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.113985    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.114155 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.114089    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.115735 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.115706    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.115830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.115784    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.120295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.120273    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxr2\" (UniqueName: \"kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.120419 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.120339    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk5xq\" (UniqueName: \"kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.226078 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.226037    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:01.271978 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.271941    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:01.503092 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.503055    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"]
Nov 01 20:56:01.505848 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:56:01.505815    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c396b0_8a07_4835_b5d1_2e104f231242.slice/crio-d386f3492857d972fa80687d3b2690193586c6ecdf98840c8a491d4631f6d673 WatchSource:0}: Error finding container d386f3492857d972fa80687d3b2690193586c6ecdf98840c8a491d4631f6d673: Status 404 returned error can't find the container with id d386f3492857d972fa80687d3b2690193586c6ecdf98840c8a491d4631f6d673
Nov 01 20:56:01.507275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.507256    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:56:01.520295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.520264    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch" event={"ID":"49c396b0-8a07-4835-b5d1-2e104f231242","Type":"ContainerStarted","Data":"d386f3492857d972fa80687d3b2690193586c6ecdf98840c8a491d4631f6d673"}
Nov 01 20:56:01.567930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.567902    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"]
Nov 01 20:56:01.571101 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:56:01.571072    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf18f866_61e9_428a_a296_3873344cc278.slice/crio-a7d7ec5201c2b81650bbd15944bd3cf95461323e5156d5ae077efa436411e9eb WatchSource:0}: Error finding container a7d7ec5201c2b81650bbd15944bd3cf95461323e5156d5ae077efa436411e9eb: Status 404 returned error can't find the container with id a7d7ec5201c2b81650bbd15944bd3cf95461323e5156d5ae077efa436411e9eb
Nov 01 20:56:01.658335 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:01.658302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:02.262523 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.262483    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:02.528054 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.528011    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch" event={"ID":"49c396b0-8a07-4835-b5d1-2e104f231242","Type":"ContainerStarted","Data":"4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed"}
Nov 01 20:56:02.528248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.528084    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:02.529641 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.529617    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" event={"ID":"af18f866-61e9-428a-a296-3873344cc278","Type":"ContainerStarted","Data":"82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965"}
Nov 01 20:56:02.529782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.529693    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" event={"ID":"af18f866-61e9-428a-a296-3873344cc278","Type":"ContainerStarted","Data":"a7d7ec5201c2b81650bbd15944bd3cf95461323e5156d5ae077efa436411e9eb"}
Nov 01 20:56:02.529782 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.529722    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:02.545458 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.545412    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch" podStartSLOduration=1.9632422109999998 podStartE2EDuration="2.545401814s" totalImagesPullingTime="582.159603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:02.543713033 +0000 UTC m=+3407.959345509" watchObservedRunningTime="2026-11-01 20:56:02.545401814 +0000 UTC m=+3407.961034285"
Nov 01 20:56:02.559911 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.559859    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" podStartSLOduration=2.559842124 podStartE2EDuration="2.559842124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:02.557698257 +0000 UTC m=+3407.973330730" watchObservedRunningTime="2026-11-01 20:56:02.559842124 +0000 UTC m=+3407.975474598"
Nov 01 20:56:02.656756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.656722    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:56:02.938715 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:02.938202    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:03.670378 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:03.664188    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:04.257871 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:04.257773    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:04.863351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:04.863302    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:05.451403 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:05.451361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:06.063774 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:06.063721    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:06.648550 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:06.648512    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:07.341905 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:07.341872    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:07.957672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:07.956198    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:08.539167 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:08.539128    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:56:08.548028 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:08.547999    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:09.313981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:09.313887    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:10.143003 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:10.142956    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:10.742791 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:10.742756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:11.357263 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:11.357224    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:12.038862 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.038822    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:12.609346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.609315    6479 generic.go:356] "Generic (PLEG): container finished" podID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerID="3dd95819de66cde32d437d5c2249369e9ee68342b9e8199d1f97b951025f952f" exitCode=0
Nov 01 20:56:12.609947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.609395    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" event={"ID":"79c95c85-ec0e-4250-9f99-2fda96b2c327","Type":"ContainerDied","Data":"3dd95819de66cde32d437d5c2249369e9ee68342b9e8199d1f97b951025f952f"}
Nov 01 20:56:12.609947 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.609935    6479 scope.go:122] "RemoveContainer" containerID="3dd95819de66cde32d437d5c2249369e9ee68342b9e8199d1f97b951025f952f"
Nov 01 20:56:12.611192 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.611162    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:56:12.611282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.611229    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:56:12.611282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.611261    6479 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 20:56:12.774166 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:12.774077    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:13.269776 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.269635    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:13.536885 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.536797    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:56:13.616427 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.616393    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" event={"ID":"79c95c85-ec0e-4250-9f99-2fda96b2c327","Type":"ContainerStarted","Data":"5273f35fb3215b48124fefd352cb33645392e674c2f7c58f2a6012cde2dad700"}
Nov 01 20:56:13.948972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.948941    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:56:13.950152 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.950124    6479 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:56:13.950282 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.950174    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 01 20:56:13.977222 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:13.977185    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:14.509479 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:14.509429    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:56:14.520273 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:14.520231    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:14.635665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:14.635600    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:14.948987 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:14.948949    6479 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:56:14.949194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:14.949018    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 01 20:56:15.078068 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.078029    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:15.177913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.177880    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:15.277593 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.277508    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:15.382549 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.382514    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:15.936200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.936168    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:15.948898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.948869    6479 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:56:15.949017 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:15.948922    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 01 20:56:16.173210 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:16.173174    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:16.820465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:16.820431    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:16.949376 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:16.949339    6479 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:56:16.949972 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:16.949392    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 01 20:56:17.545540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:17.545497    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:17.752696 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:17.752660    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:56:17.949346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:17.949305    6479 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:56:17.949545 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:17.949373    6479 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 01 20:56:18.362168 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:18.362041    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:18.953495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:18.953457    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:19.105518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:19.105487    6479 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:56:19.107440 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:19.107407    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:56:19.551368 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:19.551336    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:20.164845 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:20.164807    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:20.761701 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:20.761671    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:21.356077 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:21.356031    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:21.958979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:21.958942    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:22.558617 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:22.558572    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:23.156179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:23.156142    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:23.741739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:23.741641    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:24.314142 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:24.314110    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:24.417137 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:24.417100    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:24.951547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:24.951439    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:25.549779 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:25.549722    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:26.150388 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:26.150342    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:26.863153 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:26.863115    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:27.453308 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:27.453266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:28.050291 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:28.050244    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:28.648672 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:28.648612    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:29.169366 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:56:29.169332    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:56:29.244206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:29.244171    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:29.850622 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:29.850589    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:30.443501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:30.443470    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:31.047792 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:31.047756    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:31.741162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:31.741126    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:32.422318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:32.422250    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:33.158708 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:33.158665    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:33.613386 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:33.613351    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:56:33.832558 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:33.832519    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:34.550913 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:34.550878    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:35.154578 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:35.154546    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:35.761413 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:35.761379    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:36.350892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:36.350862    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:37.048534 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:37.048495    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:37.642601 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:37.642567    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:38.247083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:38.247040    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:38.861710 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:38.861674    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:39.443737 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:39.443698    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:40.056636 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:40.056603    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:40.649747 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:40.649713    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:41.242663 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:41.242610    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:41.851016 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:41.850983    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:42.443174 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:42.443135    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:43.047217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:43.047177    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:43.647777 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:43.647744    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:44.247536 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:44.247501    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:44.850839 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:44.850801    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:45.443727 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:45.443691    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:46.050577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:46.050541    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:46.656296 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:46.656260    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:47.347194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:47.347155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:47.954217 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:47.954185    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:48.545311 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:48.545281    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:49.143863 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:49.143828    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:49.855371 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:49.855331    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:50.448979 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:50.448942    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:51.155241 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:51.155200    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:51.752621 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:51.752583    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:52.357130 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:52.357080    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:52.957861 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:52.957828    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:53.585963 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:53.585922    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:54.151789 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:54.151759    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:54.748467 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:54.748436    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:55.351131 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:55.351099    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:55.953342 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:55.953307    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:56.556398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:56.556343    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:57.022199 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:57.022155    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:57.157892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:57.157860    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:57.469880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:57.469798    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:57.762900 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:57.762782    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:57.849605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:57.849569    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:56:58.552825 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:58.552785    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:59.159250 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:59.159210    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:56:59.171303 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:56:59.171271    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:56:59.830238 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:56:59.830207    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:00.452259 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:00.452221    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:01.051191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:01.051158    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:01.655846 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:01.655804    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:02.147527 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:02.147493    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:02.250577 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:02.250549    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:02.851310 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:02.851266    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:02.950304 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:02.950269    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"]
Nov 01 20:57:03.002134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:03.002086    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:03.102831 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:03.102741    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:03.549146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:03.549113    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:04.000566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.000525    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"]
Nov 01 20:57:04.000830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.000797    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch" podUID="49c396b0-8a07-4835-b5d1-2e104f231242" containerName="maas-api" containerID="cri-o://4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed" gracePeriod=30
Nov 01 20:57:04.042206 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.042162    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:04.239528 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.239441    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:04.326382 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.326356    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:57:04.445930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.445899    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca\") pod \"49c396b0-8a07-4835-b5d1-2e104f231242\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") "
Nov 01 20:57:04.446139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.445939    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2\" (UniqueName: \"kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2\") pod \"49c396b0-8a07-4835-b5d1-2e104f231242\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") "
Nov 01 20:57:04.446139 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.446031    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs\") pod \"49c396b0-8a07-4835-b5d1-2e104f231242\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") "
Nov 01 20:57:04.446346 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.446162    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls\") pod \"49c396b0-8a07-4835-b5d1-2e104f231242\" (UID: \"49c396b0-8a07-4835-b5d1-2e104f231242\") "
Nov 01 20:57:04.446521 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.446335    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca" pod "49c396b0-8a07-4835-b5d1-2e104f231242" (UID: "49c396b0-8a07-4835-b5d1-2e104f231242"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:04.446760 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.446638    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c396b0-8a07-4835-b5d1-2e104f231242-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:04.449256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.448686    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls" pod "49c396b0-8a07-4835-b5d1-2e104f231242" (UID: "49c396b0-8a07-4835-b5d1-2e104f231242"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:04.449256 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.449027    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs" pod "49c396b0-8a07-4835-b5d1-2e104f231242" (UID: "49c396b0-8a07-4835-b5d1-2e104f231242"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:04.451007 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.450982    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2" pod "49c396b0-8a07-4835-b5d1-2e104f231242" (UID: "49c396b0-8a07-4835-b5d1-2e104f231242"). InnerVolumeSpecName "kube-api-access-bzxr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:04.548318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.548284    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzxr2\" (UniqueName: \"kubernetes.io/projected/49c396b0-8a07-4835-b5d1-2e104f231242-kube-api-access-bzxr2\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:04.548318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.548316    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:04.548318 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.548327    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c396b0-8a07-4835-b5d1-2e104f231242-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:04.928276 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.928241    6479 generic.go:356] "Generic (PLEG): container finished" podID="49c396b0-8a07-4835-b5d1-2e104f231242" containerID="4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed" exitCode=0
Nov 01 20:57:04.928802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.928307    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"
Nov 01 20:57:04.928802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.928319    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch" event={"ID":"49c396b0-8a07-4835-b5d1-2e104f231242","Type":"ContainerDied","Data":"4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed"}
Nov 01 20:57:04.928802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.928358    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch" event={"ID":"49c396b0-8a07-4835-b5d1-2e104f231242","Type":"ContainerDied","Data":"d386f3492857d972fa80687d3b2690193586c6ecdf98840c8a491d4631f6d673"}
Nov 01 20:57:04.928802 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.928379    6479 scope.go:122] "RemoveContainer" containerID="4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed"
Nov 01 20:57:04.937594 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.937577    6479 scope.go:122] "RemoveContainer" containerID="4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed"
Nov 01 20:57:04.937877 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:04.937853    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed\": container with ID starting with 4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed not found: ID does not exist" containerID="4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed"
Nov 01 20:57:04.937955 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.937885    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed"} err="failed to get container status \"4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed\": rpc error: code = NotFound desc = could not find container \"4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed\": container with ID starting with 4bde1752ee33e60ea53a1f81fa6d2518a16d538362ee14f3546972c6405261ed not found: ID does not exist"
Nov 01 20:57:04.944317 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.944295    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"]
Nov 01 20:57:04.947852 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:04.947832    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-gghch"]
Nov 01 20:57:05.554518 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:05.554485    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:57:06.705899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:06.705867    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c396b0-8a07-4835-b5d1-2e104f231242" path="/var/lib/kubelet/pods/49c396b0-8a07-4835-b5d1-2e104f231242/volumes"
Nov 01 20:57:07.951194 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:07.951150    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" podUID="af18f866-61e9-428a-a296-3873344cc278" containerName="payload-pre-processing" containerID="cri-o://82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965" gracePeriod=25
Nov 01 20:57:09.623466 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:09.623414    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:09.724281 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:09.724246    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:13.535209 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:13.535171    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" podUID="af18f866-61e9-428a-a296-3873344cc278" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:57:23.534956 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:23.534914    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" podUID="af18f866-61e9-428a-a296-3873344cc278" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:57:24.359864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:24.359825    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:28.201505 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.201474    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:57:28.284204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.284111    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq\" (UniqueName: \"kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq\") pod \"af18f866-61e9-428a-a296-3873344cc278\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") "
Nov 01 20:57:28.284204 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.284197    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume\") pod \"af18f866-61e9-428a-a296-3873344cc278\" (UID: \"af18f866-61e9-428a-a296-3873344cc278\") "
Nov 01 20:57:28.284571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.284546    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume" pod "af18f866-61e9-428a-a296-3873344cc278" (UID: "af18f866-61e9-428a-a296-3873344cc278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:28.286181 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.286160    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq" pod "af18f866-61e9-428a-a296-3873344cc278" (UID: "af18f866-61e9-428a-a296-3873344cc278"). InnerVolumeSpecName "kube-api-access-mk5xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:28.385441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.385407    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk5xq\" (UniqueName: \"kubernetes.io/projected/af18f866-61e9-428a-a296-3873344cc278-kube-api-access-mk5xq\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:28.385441 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:28.385435    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af18f866-61e9-428a-a296-3873344cc278-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:29.050337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.050242    6479 generic.go:356] "Generic (PLEG): container finished" podID="af18f866-61e9-428a-a296-3873344cc278" containerID="82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965" exitCode=0
Nov 01 20:57:29.050337 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.050292    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"
Nov 01 20:57:29.050598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.050334    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" event={"ID":"af18f866-61e9-428a-a296-3873344cc278","Type":"ContainerDied","Data":"82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965"}
Nov 01 20:57:29.050598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.050398    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl" event={"ID":"af18f866-61e9-428a-a296-3873344cc278","Type":"ContainerDied","Data":"a7d7ec5201c2b81650bbd15944bd3cf95461323e5156d5ae077efa436411e9eb"}
Nov 01 20:57:29.050598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.050430    6479 scope.go:122] "RemoveContainer" containerID="82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965"
Nov 01 20:57:29.059162 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.059136    6479 scope.go:122] "RemoveContainer" containerID="82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965"
Nov 01 20:57:29.059417 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:29.059393    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965\": container with ID starting with 82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965 not found: ID does not exist" containerID="82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965"
Nov 01 20:57:29.059484 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.059427    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965"} err="failed to get container status \"82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965\": rpc error: code = NotFound desc = could not find container \"82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965\": container with ID starting with 82e8eed4edea7a9998e6f20fddc8e8589b1da66116bfb3e879ce2745f1975965 not found: ID does not exist"
Nov 01 20:57:29.066059 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.066034    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"]
Nov 01 20:57:29.068363 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:29.068336    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784fsqwl"]
Nov 01 20:57:29.170150 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:29.170111    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:57:30.706340 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:30.706305    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af18f866-61e9-428a-a296-3873344cc278" path="/var/lib/kubelet/pods/af18f866-61e9-428a-a296-3873344cc278/volumes"
Nov 01 20:57:34.192465 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:34.192417    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:35.315275 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.315237    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"]
Nov 01 20:57:35.315868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.315845    6479 cpu_manager.go:414] "containerMap: removing container" podUID="af18f866-61e9-428a-a296-3873344cc278" containerName="payload-pre-processing"
Nov 01 20:57:35.315868 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.315867    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af18f866-61e9-428a-a296-3873344cc278" containerName="payload-pre-processing"
Nov 01 20:57:35.316035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.315902    6479 cpu_manager.go:414] "containerMap: removing container" podUID="49c396b0-8a07-4835-b5d1-2e104f231242" containerName="maas-api"
Nov 01 20:57:35.316035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.315907    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c396b0-8a07-4835-b5d1-2e104f231242" containerName="maas-api"
Nov 01 20:57:35.316035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.315985    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="af18f866-61e9-428a-a296-3873344cc278" containerName="payload-pre-processing"
Nov 01 20:57:35.316035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.316003    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c396b0-8a07-4835-b5d1-2e104f231242" containerName="maas-api"
Nov 01 20:57:35.321246 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.320836    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.327801 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.327778    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4e9e72-serving-cert\""
Nov 01 20:57:35.329397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.329370    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"]
Nov 01 20:57:35.355720 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.355685    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmhk\" (UniqueName: \"kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.355981 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.355953    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.356113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.356029    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.356113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.356089    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.424587 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.424553    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"]
Nov 01 20:57:35.428122 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.428100    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.437754 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.437089    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"]
Nov 01 20:57:35.438344 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.438313    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bbf535b9-dockercfg-wfxlf\""
Nov 01 20:57:35.457414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.457373    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmhk\" (UniqueName: \"kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.457598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.457439    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.457598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.457497    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.457598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.457547    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.457598 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.457586    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2n7\" (UniqueName: \"kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.457860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.457663    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.458340 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:35.458316    6479 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-serving-cert: secret "maas-api-e2e-worker-single-main-4e9e72-serving-cert" not found
Nov 01 20:57:35.458458 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:35.458394    6479 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls podName:6232891b-fbb9-4cf6-895b-619773453be4 nodeName:}" failed. No retries permitted until 2026-11-01 20:57:35.958376975 +0000 UTC m=+3501.374009439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls") pod "maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" (UID: "6232891b-fbb9-4cf6-895b-619773453be4") : secret "maas-api-e2e-worker-single-main-4e9e72-serving-cert" not found
Nov 01 20:57:35.459118 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.459079    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.460985 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.460956    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.466191 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.466165    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmhk\" (UniqueName: \"kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.532512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.532479    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e9e72\""
Nov 01 20:57:35.585717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.574244    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2n7\" (UniqueName: \"kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.585717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.574329    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.585717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.575614    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.593150 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.593118    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2n7\" (UniqueName: \"kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7\") pod \"payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.674454 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.674422    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:35.747083 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.747052    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:35.774221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.774191    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:35.978883 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.978830    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:35.981482 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:35.981456    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:36.023106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:36.023078    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"]
Nov 01 20:57:36.025460 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:57:36.025428    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddf2338_11da_45bf_aebb_d94abad2024b.slice/crio-8547595fcc735dc0029343ec72f24d43e14cd09a321f98a204077af4ef47a139 WatchSource:0}: Error finding container 8547595fcc735dc0029343ec72f24d43e14cd09a321f98a204077af4ef47a139: Status 404 returned error can't find the container with id 8547595fcc735dc0029343ec72f24d43e14cd09a321f98a204077af4ef47a139
Nov 01 20:57:36.086367 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:36.086336    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms" event={"ID":"1ddf2338-11da-45bf-aebb-d94abad2024b","Type":"ContainerStarted","Data":"8547595fcc735dc0029343ec72f24d43e14cd09a321f98a204077af4ef47a139"}
Nov 01 20:57:36.199397 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:36.199358    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:36.240889 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:36.240808    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:36.533253 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:36.533225    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"]
Nov 01 20:57:36.536051 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:57:36.536013    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6232891b_fbb9_4cf6_895b_619773453be4.slice/crio-c81b0a9961e427a5560c373142fbf38203899bac051d7fc11944776bdb4a93ce WatchSource:0}: Error finding container c81b0a9961e427a5560c373142fbf38203899bac051d7fc11944776bdb4a93ce: Status 404 returned error can't find the container with id c81b0a9961e427a5560c373142fbf38203899bac051d7fc11944776bdb4a93ce
Nov 01 20:57:36.927365 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:36.927263    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:37.095374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.094886    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms" event={"ID":"1ddf2338-11da-45bf-aebb-d94abad2024b","Type":"ContainerStarted","Data":"86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c"}
Nov 01 20:57:37.095374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.095109    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:37.096750 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.096726    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" event={"ID":"6232891b-fbb9-4cf6-895b-619773453be4","Type":"ContainerStarted","Data":"ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138"}
Nov 01 20:57:37.096877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.096770    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" event={"ID":"6232891b-fbb9-4cf6-895b-619773453be4","Type":"ContainerStarted","Data":"c81b0a9961e427a5560c373142fbf38203899bac051d7fc11944776bdb4a93ce"}
Nov 01 20:57:37.096877 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.096863    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:37.112061 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.112010    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms" podStartSLOduration=2.1119963999999998 podStartE2EDuration="2.1119964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:37.110021212 +0000 UTC m=+3502.525653687" watchObservedRunningTime="2026-11-01 20:57:37.1119964 +0000 UTC m=+3502.527628869"
Nov 01 20:57:37.131313 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.131267    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" podStartSLOduration=1.642913209 podStartE2EDuration="2.131255127s" totalImagesPullingTime="488.341918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:37.130105774 +0000 UTC m=+3502.545738247" watchObservedRunningTime="2026-11-01 20:57:37.131255127 +0000 UTC m=+3502.546887600"
Nov 01 20:57:37.447814 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.447783    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:57:37.636035 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:37.635980    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:38.253878 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:38.253849    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:38.449571 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:38.449532    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:39.169866 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:39.169834    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:39.475411 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:39.475344    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:39.749010 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:39.748895    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:40.254717 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:40.254675    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:57:40.453771 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:40.453734    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:41.250869 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:41.250834    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:41.842414 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:41.842379    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:42.447398 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:42.447361    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:43.051977 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:43.051941    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:43.115218 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:43.115186    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:43.649933 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:43.649888    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:44.352144 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:44.352106    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:44.960327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:44.960287    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:45.251818 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:45.251737    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:45.352982 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:45.352947    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:45.558417 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:45.558325    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:46.166415 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:46.166375    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:46.272745 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:46.272696    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:46.752106 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:46.752061    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:47.372353 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:47.372324    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:48.051233 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:48.051202    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:48.119316 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:48.118970    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:48.648925 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:48.648890    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:49.252158 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:49.252121    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:49.860959 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:49.860924    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:50.513620 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:50.513571    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:51.052396 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:51.052365    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:51.651349 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:51.651314    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:52.267134 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:52.267097    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:52.860788 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:52.860743    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:53.461179 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:53.461139    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.049958 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.049925    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.312124 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.312045    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"]
Nov 01 20:57:54.361432 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.361390    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.462770 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.462733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.524971 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.524924    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"]
Nov 01 20:57:54.525220 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.525192    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" podUID="6232891b-fbb9-4cf6-895b-619773453be4" containerName="maas-api" containerID="cri-o://ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138" gracePeriod=30
Nov 01 20:57:54.561766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.561733    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.684146 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.684114    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.754859 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.754819    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:54.811860 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.810672    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:54.901995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.901880    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs\") pod \"6232891b-fbb9-4cf6-895b-619773453be4\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") "
Nov 01 20:57:54.901995 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.901945    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk\" (UniqueName: \"kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk\") pod \"6232891b-fbb9-4cf6-895b-619773453be4\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") "
Nov 01 20:57:54.902230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.902011    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca\") pod \"6232891b-fbb9-4cf6-895b-619773453be4\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") "
Nov 01 20:57:54.902230 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.902085    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\") pod \"6232891b-fbb9-4cf6-895b-619773453be4\" (UID: \"6232891b-fbb9-4cf6-895b-619773453be4\") "
Nov 01 20:57:54.902476 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.902451    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca" pod "6232891b-fbb9-4cf6-895b-619773453be4" (UID: "6232891b-fbb9-4cf6-895b-619773453be4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:54.904198 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.904170    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk" pod "6232891b-fbb9-4cf6-895b-619773453be4" (UID: "6232891b-fbb9-4cf6-895b-619773453be4"). InnerVolumeSpecName "kube-api-access-mxmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:54.904293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.904238    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls" pod "6232891b-fbb9-4cf6-895b-619773453be4" (UID: "6232891b-fbb9-4cf6-895b-619773453be4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:54.904352 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:54.904334    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs" pod "6232891b-fbb9-4cf6-895b-619773453be4" (UID: "6232891b-fbb9-4cf6-895b-619773453be4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:55.003196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.003154    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:55.003196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.003186    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6232891b-fbb9-4cf6-895b-619773453be4-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:55.003196 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.003202    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmhk\" (UniqueName: \"kubernetes.io/projected/6232891b-fbb9-4cf6-895b-619773453be4-kube-api-access-mxmhk\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:55.003462 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.003217    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6232891b-fbb9-4cf6-895b-619773453be4-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:55.029495 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.029460    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"]
Nov 01 20:57:55.029756 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.029729    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy" containerID="cri-o://5273f35fb3215b48124fefd352cb33645392e674c2f7c58f2a6012cde2dad700" gracePeriod=30
Nov 01 20:57:55.069680 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.069632    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:57:55.189930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.189857    6479 generic.go:356] "Generic (PLEG): container finished" podID="6232891b-fbb9-4cf6-895b-619773453be4" containerID="ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138" exitCode=0
Nov 01 20:57:55.189930 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.189905    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"
Nov 01 20:57:55.190373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.189943    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" event={"ID":"6232891b-fbb9-4cf6-895b-619773453be4","Type":"ContainerDied","Data":"ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138"}
Nov 01 20:57:55.190373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.190007    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb" event={"ID":"6232891b-fbb9-4cf6-895b-619773453be4","Type":"ContainerDied","Data":"c81b0a9961e427a5560c373142fbf38203899bac051d7fc11944776bdb4a93ce"}
Nov 01 20:57:55.190373 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.190038    6479 scope.go:122] "RemoveContainer" containerID="ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138"
Nov 01 20:57:55.199429 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.199411    6479 scope.go:122] "RemoveContainer" containerID="ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138"
Nov 01 20:57:55.199750 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:55.199730    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138\": container with ID starting with ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138 not found: ID does not exist" containerID="ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138"
Nov 01 20:57:55.199830 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.199759    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138"} err="failed to get container status \"ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138\": rpc error: code = NotFound desc = could not find container \"ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138\": container with ID starting with ea423c47f74bb9fd79cd3dee7ff87b0e062747903af41bc50381d87aa1e35138 not found: ID does not exist"
Nov 01 20:57:55.218046 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.218016    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"]
Nov 01 20:57:55.226753 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.226728    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e9e72-5bdbd798fc-cdkfb"]
Nov 01 20:57:55.931216 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:55.931171    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:57:56.710799 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:56.710751    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6232891b-fbb9-4cf6-895b-619773453be4" path="/var/lib/kubelet/pods/6232891b-fbb9-4cf6-895b-619773453be4/volumes"
Nov 01 20:57:59.169909 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:57:59.169872    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:57:59.313391 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.313351    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms" podUID="1ddf2338-11da-45bf-aebb-d94abad2024b" containerName="payload-pre-processing" containerID="cri-o://86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c" gracePeriod=25
Nov 01 20:57:59.565177 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.565152    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:57:59.650700 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.650628    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7\" (UniqueName: \"kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7\") pod \"1ddf2338-11da-45bf-aebb-d94abad2024b\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") "
Nov 01 20:57:59.650901 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.650758    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume\") pod \"1ddf2338-11da-45bf-aebb-d94abad2024b\" (UID: \"1ddf2338-11da-45bf-aebb-d94abad2024b\") "
Nov 01 20:57:59.651113 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.651090    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume" pod "1ddf2338-11da-45bf-aebb-d94abad2024b" (UID: "1ddf2338-11da-45bf-aebb-d94abad2024b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:59.652739 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.652714    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7" pod "1ddf2338-11da-45bf-aebb-d94abad2024b" (UID: "1ddf2338-11da-45bf-aebb-d94abad2024b"). InnerVolumeSpecName "kube-api-access-5s2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:59.751880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.751847    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s2n7\" (UniqueName: \"kubernetes.io/projected/1ddf2338-11da-45bf-aebb-d94abad2024b-kube-api-access-5s2n7\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:59.751880 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:57:59.751875    6479 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ddf2338-11da-45bf-aebb-d94abad2024b-plugins-config-volume\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.216916 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.216881    6479 generic.go:356] "Generic (PLEG): container finished" podID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerID="5273f35fb3215b48124fefd352cb33645392e674c2f7c58f2a6012cde2dad700" exitCode=0
Nov 01 20:58:00.217320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.216962    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" event={"ID":"79c95c85-ec0e-4250-9f99-2fda96b2c327","Type":"ContainerDied","Data":"5273f35fb3215b48124fefd352cb33645392e674c2f7c58f2a6012cde2dad700"}
Nov 01 20:58:00.217320 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.217041    6479 scope.go:122] "RemoveContainer" containerID="3dd95819de66cde32d437d5c2249369e9ee68342b9e8199d1f97b951025f952f"
Nov 01 20:58:00.218966 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.218943    6479 generic.go:356] "Generic (PLEG): container finished" podID="1ddf2338-11da-45bf-aebb-d94abad2024b" containerID="86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c" exitCode=0
Nov 01 20:58:00.219087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.218986    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"
Nov 01 20:58:00.219087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.219008    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms" event={"ID":"1ddf2338-11da-45bf-aebb-d94abad2024b","Type":"ContainerDied","Data":"86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c"}
Nov 01 20:58:00.219087 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.219055    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms" event={"ID":"1ddf2338-11da-45bf-aebb-d94abad2024b","Type":"ContainerDied","Data":"8547595fcc735dc0029343ec72f24d43e14cd09a321f98a204077af4ef47a139"}
Nov 01 20:58:00.232547 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.232525    6479 scope.go:122] "RemoveContainer" containerID="86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c"
Nov 01 20:58:00.241136 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.241111    6479 scope.go:122] "RemoveContainer" containerID="86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c"
Nov 01 20:58:00.241499 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:58:00.241466    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c\": container with ID starting with 86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c not found: ID does not exist" containerID="86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c"
Nov 01 20:58:00.241606 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.241510    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c"} err="failed to get container status \"86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c\": rpc error: code = NotFound desc = could not find container \"86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c\": container with ID starting with 86a2afb8d8b4d6ea78e5fe7dac04c92c4f45fa6ce95fb912d9898186a06f1c4c not found: ID does not exist"
Nov 01 20:58:00.243338 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.243314    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"]
Nov 01 20:58:00.245713 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.245685    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e9e72-56784nkvms"]
Nov 01 20:58:00.278816 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.278797    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:58:00.357602 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357572    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.357766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357637    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.357766 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357687    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.357898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357808    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.357898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357855    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.357898 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357881    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:00.358065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357888    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.358065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357929    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:00.358065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.357966    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.358065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358045    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:00.358065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358056    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:00.358065 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358063    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.358383 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358117    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert\") pod \"79c95c85-ec0e-4250-9f99-2fda96b2c327\" (UID: \"79c95c85-ec0e-4250-9f99-2fda96b2c327\") "
Nov 01 20:58:00.358448 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358427    6479 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-credential-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.358511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358446    6479 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-data\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.358511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358460    6479 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-socket\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.358511 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358475    6479 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-workload-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.358639 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.358531    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:00.360111 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.360083    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "kube-api-access-cg9vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:00.360327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.360303    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:58:00.360457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.360435    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:58:00.360457 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.360442    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token" pod "79c95c85-ec0e-4250-9f99-2fda96b2c327" (UID: "79c95c85-ec0e-4250-9f99-2fda96b2c327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:00.459626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.459590    6479 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c95c85-ec0e-4250-9f99-2fda96b2c327-istiod-ca-cert\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.459626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.459621    6479 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-envoy\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.459626 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.459631    6479 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-podinfo\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.459892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.459640    6479 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-istio-token\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.459892 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.459677    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg9vt\" (UniqueName: \"kubernetes.io/projected/79c95c85-ec0e-4250-9f99-2fda96b2c327-kube-api-access-cg9vt\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:00.707128 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:00.707095    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddf2338-11da-45bf-aebb-d94abad2024b" path="/var/lib/kubelet/pods/1ddf2338-11da-45bf-aebb-d94abad2024b/volumes"
Nov 01 20:58:01.224806 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:01.224768    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"
Nov 01 20:58:01.225293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:01.224805    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr" event={"ID":"79c95c85-ec0e-4250-9f99-2fda96b2c327","Type":"ContainerDied","Data":"1ef3b47e6557e87d122a829c8d1a65f5caa68c1b3508e76892e1ed820902d005"}
Nov 01 20:58:01.225293 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:01.224880    6479 scope.go:122] "RemoveContainer" containerID="5273f35fb3215b48124fefd352cb33645392e674c2f7c58f2a6012cde2dad700"
Nov 01 20:58:01.241875 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:01.241844    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"]
Nov 01 20:58:01.245512 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:01.245486    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e9e72-openshift-default-fb6f65d77-rjnxr"]
Nov 01 20:58:02.706486 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:02.706451    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" path="/var/lib/kubelet/pods/79c95c85-ec0e-4250-9f99-2fda96b2c327/volumes"
Nov 01 20:58:03.049919 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:03.049824    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:58:07.047858 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:07.047822    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:58:11.102115 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:11.102076    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:58:14.345671 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:14.345397    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:58:24.401562 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:24.401528    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:58:27.528351 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:27.528312    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:58:28.630207 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630174    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"]
Nov 01 20:58:28.630768 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630747    6479 cpu_manager.go:414] "containerMap: removing container" podUID="1ddf2338-11da-45bf-aebb-d94abad2024b" containerName="payload-pre-processing"
Nov 01 20:58:28.630864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630770    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddf2338-11da-45bf-aebb-d94abad2024b" containerName="payload-pre-processing"
Nov 01 20:58:28.630864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630816    6479 cpu_manager.go:414] "containerMap: removing container" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy"
Nov 01 20:58:28.630864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630825    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy"
Nov 01 20:58:28.630864 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630856    6479 cpu_manager.go:414] "containerMap: removing container" podUID="6232891b-fbb9-4cf6-895b-619773453be4" containerName="maas-api"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630869    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6232891b-fbb9-4cf6-895b-619773453be4" containerName="maas-api"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630891    6479 cpu_manager.go:414] "containerMap: removing container" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630902    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.630997    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddf2338-11da-45bf-aebb-d94abad2024b" containerName="payload-pre-processing"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.631017    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.631029    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="6232891b-fbb9-4cf6-895b-619773453be4" containerName="maas-api"
Nov 01 20:58:28.631089 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.631039    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c95c85-ec0e-4250-9f99-2fda96b2c327" containerName="istio-proxy"
Nov 01 20:58:28.635683 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.635663    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.644684 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.644631    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"]
Nov 01 20:58:28.729540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.729504    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656671-ee1f-4343-9ea4-3874fcf3be28-openshift-service-ca\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.729540 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.729542    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac656671-ee1f-4343-9ea4-3874fcf3be28-maas-api-tls\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.729803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.729657    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac656671-ee1f-4343-9ea4-3874fcf3be28-metrics-certs\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.729803 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.729704    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwlf\" (UniqueName: \"kubernetes.io/projected/ac656671-ee1f-4343-9ea4-3874fcf3be28-kube-api-access-nbwlf\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.831200 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.831156    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656671-ee1f-4343-9ea4-3874fcf3be28-openshift-service-ca\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.831404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.831207    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac656671-ee1f-4343-9ea4-3874fcf3be28-maas-api-tls\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.831404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.831257    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac656671-ee1f-4343-9ea4-3874fcf3be28-metrics-certs\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.831404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.831288    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwlf\" (UniqueName: \"kubernetes.io/projected/ac656671-ee1f-4343-9ea4-3874fcf3be28-kube-api-access-nbwlf\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.832248 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.832222    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656671-ee1f-4343-9ea4-3874fcf3be28-openshift-service-ca\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.833899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.833872    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac656671-ee1f-4343-9ea4-3874fcf3be28-maas-api-tls\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.834266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.834240    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac656671-ee1f-4343-9ea4-3874fcf3be28-metrics-certs\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.838694 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.838672    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwlf\" (UniqueName: \"kubernetes.io/projected/ac656671-ee1f-4343-9ea4-3874fcf3be28-kube-api-access-nbwlf\") pod \"maas-api-6b5986f9c5-qn2kx\" (UID: \"ac656671-ee1f-4343-9ea4-3874fcf3be28\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:28.951471 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:28.951436    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:29.170221 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:58:29.170179    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:58:29.229264 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:29.229150    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"]
Nov 01 20:58:29.231860 ip-10-0-133-52 kubenswrapper[6479]: W1101 20:58:29.231829    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac656671_ee1f_4343_9ea4_3874fcf3be28.slice/crio-94ad6c6a454d5587d9118e44d6a998a5d6ccca25ba8eb7f934c2d607faf218ef WatchSource:0}: Error finding container 94ad6c6a454d5587d9118e44d6a998a5d6ccca25ba8eb7f934c2d607faf218ef: Status 404 returned error can't find the container with id 94ad6c6a454d5587d9118e44d6a998a5d6ccca25ba8eb7f934c2d607faf218ef
Nov 01 20:58:29.488514 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:29.488423    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx" event={"ID":"ac656671-ee1f-4343-9ea4-3874fcf3be28","Type":"ContainerStarted","Data":"94ad6c6a454d5587d9118e44d6a998a5d6ccca25ba8eb7f934c2d607faf218ef"}
Nov 01 20:58:30.494899 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:30.494856    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx" event={"ID":"ac656671-ee1f-4343-9ea4-3874fcf3be28","Type":"ContainerStarted","Data":"cdc1e67200479c7d39e55fc38e7fbf1cdb3dea7a16ae71456500a26d328c72ae"}
Nov 01 20:58:30.495323 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:30.494981    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:30.511941 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:30.511605    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx" podStartSLOduration=2.062151381 podStartE2EDuration="2.511588535s" totalImagesPullingTime="449.437154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:30.511590738 +0000 UTC m=+3555.927223248" watchObservedRunningTime="2026-11-01 20:58:30.511588535 +0000 UTC m=+3555.927221008"
Nov 01 20:58:36.504983 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:36.504947    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-qn2kx"
Nov 01 20:58:36.644501 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:36.644464    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:58:43.926184 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:43.926144    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"]
Nov 01 20:58:43.926614 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:43.926385    6479 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" podUID="58a63c17-402b-4604-bf9c-e4184b62c283" containerName="maas-api" containerID="cri-o://f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792" gracePeriod=30
Nov 01 20:58:43.984844 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:43.984795    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" podUID="58a63c17-402b-4604-bf9c-e4184b62c283" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.24:8443/health\": dial tcp 10.134.0.24:8443: connect: connection refused"
Nov 01 20:58:45.179325 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.179301    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:58:45.191347 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.191321    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca\") pod \"58a63c17-402b-4604-bf9c-e4184b62c283\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") "
Nov 01 20:58:45.191436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.191379    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn\" (UniqueName: \"kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn\") pod \"58a63c17-402b-4604-bf9c-e4184b62c283\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") "
Nov 01 20:58:45.191436 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.191426    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls\") pod \"58a63c17-402b-4604-bf9c-e4184b62c283\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") "
Nov 01 20:58:45.191553 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.191455    6479 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs\") pod \"58a63c17-402b-4604-bf9c-e4184b62c283\" (UID: \"58a63c17-402b-4604-bf9c-e4184b62c283\") "
Nov 01 20:58:45.191699 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.191666    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca" pod "58a63c17-402b-4604-bf9c-e4184b62c283" (UID: "58a63c17-402b-4604-bf9c-e4184b62c283"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:45.193556 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.193533    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs" pod "58a63c17-402b-4604-bf9c-e4184b62c283" (UID: "58a63c17-402b-4604-bf9c-e4184b62c283"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:58:45.193665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.193578    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn" pod "58a63c17-402b-4604-bf9c-e4184b62c283" (UID: "58a63c17-402b-4604-bf9c-e4184b62c283"). InnerVolumeSpecName "kube-api-access-jj2xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:45.193665 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.193590    6479 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls" pod "58a63c17-402b-4604-bf9c-e4184b62c283" (UID: "58a63c17-402b-4604-bf9c-e4184b62c283"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:58:45.292896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.292823    6479 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-metrics-certs\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:45.292896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.292854    6479 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a63c17-402b-4604-bf9c-e4184b62c283-openshift-service-ca\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:45.292896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.292867    6479 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2xn\" (UniqueName: \"kubernetes.io/projected/58a63c17-402b-4604-bf9c-e4184b62c283-kube-api-access-jj2xn\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:45.292896 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.292881    6479 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a63c17-402b-4604-bf9c-e4184b62c283-maas-api-tls\") on node \"ip-10-0-133-52.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:45.614266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.614187    6479 generic.go:356] "Generic (PLEG): container finished" podID="58a63c17-402b-4604-bf9c-e4184b62c283" containerID="f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792" exitCode=0
Nov 01 20:58:45.614266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.614212    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" event={"ID":"58a63c17-402b-4604-bf9c-e4184b62c283","Type":"ContainerDied","Data":"f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792"}
Nov 01 20:58:45.614266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.614232    6479 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"
Nov 01 20:58:45.614266 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.614268    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq" event={"ID":"58a63c17-402b-4604-bf9c-e4184b62c283","Type":"ContainerDied","Data":"edf54fe8175a95ec6882d84193b614ee36184cae29a45deae674fcf6d056c767"}
Nov 01 20:58:45.614533 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.614291    6479 scope.go:122] "RemoveContainer" containerID="f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792"
Nov 01 20:58:45.623566 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.623548    6479 scope.go:122] "RemoveContainer" containerID="f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792"
Nov 01 20:58:45.623857 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:58:45.623838    6479 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792\": container with ID starting with f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792 not found: ID does not exist" containerID="f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792"
Nov 01 20:58:45.623949 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.623869    6479 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792"} err="failed to get container status \"f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792\": rpc error: code = NotFound desc = could not find container \"f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792\": container with ID starting with f19b33d2cf0f069284c279580dec632fae98543650ed2fc3985e58d4a5347792 not found: ID does not exist"
Nov 01 20:58:45.634297 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.634274    6479 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"]
Nov 01 20:58:45.636605 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:45.636574    6479 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-qlxbq"]
Nov 01 20:58:46.706221 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:58:46.706183    6479 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a63c17-402b-4604-bf9c-e4184b62c283" path="/var/lib/kubelet/pods/58a63c17-402b-4604-bf9c-e4184b62c283/volumes"
Nov 01 20:58:59.169343 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:58:59.169305    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:59:00.502295 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:00.502241    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:59:06.526470 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:06.526425    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:59:09.529854 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:09.529813    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:59:24.503374 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:24.503341    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 20:59:26.050404 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:26.050367    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:59:29.184072 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:59:29.184010    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 20:59:31.310709 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:31.310663    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:59:52.904327 ip-10-0-133-52 kubenswrapper[6479]: I1101 20:59:52.904290    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 20:59:59.168919 ip-10-0-133-52 kubenswrapper[6479]: E1101 20:59:59.168881    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 21:00:03.154398 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:03.154357    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 21:00:08.343670 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:08.343039    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 21:00:14.200855 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:14.200813    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 21:00:24.490457 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:24.490420    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 21:00:29.171085 ip-10-0-133-52 kubenswrapper[6479]: E1101 21:00:29.171040    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 21:00:32.743196 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:32.743155    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 21:00:35.821581 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:35.821538    6479 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-c6qrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:00:35.822228 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:35.822191    6479 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-c6qrt" podUID="c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:00:42.944146 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:00:42.944104    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v5k66"]
Nov 01 21:00:59.169602 ip-10-0-133-52 kubenswrapper[6479]: E1101 21:00:59.169521    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 21:01:01.481145 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:01.481110    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-trh22_eff10696-0cfb-4433-ad91-3d0e817fd274/manager/0.log"
Nov 01 21:01:03.601367 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:03.601335    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b5986f9c5-qn2kx_ac656671-ee1f-4343-9ea4-3874fcf3be28/maas-api/0.log"
Nov 01 21:01:08.024499 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:08.024465    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dvwgc_c59e357a-a3be-4269-9cc3-de751b07d38a/manager/0.log"
Nov 01 21:01:08.259034 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:08.259001    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v5k66_8c6137ad-fb29-4667-be40-173018e00d94/limitador/0.log"
Nov 01 21:01:09.651146 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:09.651116    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k44gg_c60b32e8-0368-46cc-9385-ba469e82c957/discovery/0.log"
Nov 01 21:01:09.817760 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:09.817732    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-wwjm6_ff9972fd-5487-4ab7-b0dd-0962e84e88c9/kube-auth-proxy/0.log"
Nov 01 21:01:10.163898 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:10.163852    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-989zh_4e8677a8-ebe4-4a37-9a4e-e43551a314ff/istio-proxy/0.log"
Nov 01 21:01:10.532545 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:10.532510    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-c6qrt_c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5/istio-proxy/0.log"
Nov 01 21:01:10.896186 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:10.896116    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jsxtq_838f2059-c8d7-4b2c-95e9-a9a31c87683c/istio-proxy/0.log"
Nov 01 21:01:11.812979 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:11.812948    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s4pct_020b4fc0-677d-4ec2-ac2d-bdd1379e7382/istio-proxy/0.log"
Nov 01 21:01:12.150313 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:12.150229    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7f75799d9b-w7752_d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b/payload-pre-processing/0.log"
Nov 01 21:01:12.314826 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:12.314796    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5ff9ddc8d7-c89gd_7c77aedf-651e-490c-a0e7-31afeacf0102/payload-processing/0.log"
Nov 01 21:01:13.936826 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:13.936793    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fhmpr_159ab949-f5cd-452a-94ca-89c96023149d/main/0.log"
Nov 01 21:01:14.284095 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:14.284059    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t7zf2_bdcf6c19-7418-4ed0-ac7c-9abaed9df5b3/main/0.log"
Nov 01 21:01:14.448225 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:14.448173    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bl4xv_ac54c9bd-b9df-47d3-b566-73a43f6263e5/main/0.log"
Nov 01 21:01:24.274236 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.274176    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 21:01:24.370247 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.370211    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z9gbm/must-gather-85gd8"]
Nov 01 21:01:24.370761 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.370730    6479 cpu_manager.go:414] "containerMap: removing container" podUID="58a63c17-402b-4604-bf9c-e4184b62c283" containerName="maas-api"
Nov 01 21:01:24.370761 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.370762    6479 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a63c17-402b-4604-bf9c-e4184b62c283" containerName="maas-api"
Nov 01 21:01:24.370912 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.370847    6479 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a63c17-402b-4604-bf9c-e4184b62c283" containerName="maas-api"
Nov 01 21:01:24.373964 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.373945    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.380052 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.380019    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z9gbm\"/\"kube-root-ca.crt\""
Nov 01 21:01:24.382001 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.381770    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/must-gather-85gd8"]
Nov 01 21:01:24.382934 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.382910    6479 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z9gbm\"/\"openshift-service-ca.crt\""
Nov 01 21:01:24.383217 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.383197    6479 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z9gbm\"/\"default-dockercfg-fndqt\""
Nov 01 21:01:24.397243 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.397196    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 21:01:24.432995 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.432959    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9zs\" (UniqueName: \"kubernetes.io/projected/98ab4575-379b-45f5-bc42-9f0676cdccf9-kube-api-access-xm9zs\") pod \"must-gather-85gd8\" (UID: \"98ab4575-379b-45f5-bc42-9f0676cdccf9\") " pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.433167 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.433029    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98ab4575-379b-45f5-bc42-9f0676cdccf9-must-gather-output\") pod \"must-gather-85gd8\" (UID: \"98ab4575-379b-45f5-bc42-9f0676cdccf9\") " pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.534298 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.534208    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98ab4575-379b-45f5-bc42-9f0676cdccf9-must-gather-output\") pod \"must-gather-85gd8\" (UID: \"98ab4575-379b-45f5-bc42-9f0676cdccf9\") " pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.534459 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.534313    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9zs\" (UniqueName: \"kubernetes.io/projected/98ab4575-379b-45f5-bc42-9f0676cdccf9-kube-api-access-xm9zs\") pod \"must-gather-85gd8\" (UID: \"98ab4575-379b-45f5-bc42-9f0676cdccf9\") " pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.534596 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.534573    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98ab4575-379b-45f5-bc42-9f0676cdccf9-must-gather-output\") pod \"must-gather-85gd8\" (UID: \"98ab4575-379b-45f5-bc42-9f0676cdccf9\") " pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.536488 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.536440    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 21:01:24.542375 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.542349    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9zs\" (UniqueName: \"kubernetes.io/projected/98ab4575-379b-45f5-bc42-9f0676cdccf9-kube-api-access-xm9zs\") pod \"must-gather-85gd8\" (UID: \"98ab4575-379b-45f5-bc42-9f0676cdccf9\") " pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.706854 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.706826    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/must-gather-85gd8"
Nov 01 21:01:24.981139 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.981052    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/must-gather-85gd8"]
Nov 01 21:01:24.983831 ip-10-0-133-52 kubenswrapper[6479]: W1101 21:01:24.983799    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ab4575_379b_45f5_bc42_9f0676cdccf9.slice/crio-3f150fb722a1d2b506fbf7f2bbcf6245e57c866a51633c798c385496fd4e9d19 WatchSource:0}: Error finding container 3f150fb722a1d2b506fbf7f2bbcf6245e57c866a51633c798c385496fd4e9d19: Status 404 returned error can't find the container with id 3f150fb722a1d2b506fbf7f2bbcf6245e57c866a51633c798c385496fd4e9d19
Nov 01 21:01:24.985757 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:24.985736    6479 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:01:25.517451 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:25.517412    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/must-gather-85gd8" event={"ID":"98ab4575-379b-45f5-bc42-9f0676cdccf9","Type":"ContainerStarted","Data":"3f150fb722a1d2b506fbf7f2bbcf6245e57c866a51633c798c385496fd4e9d19"}
Nov 01 21:01:26.528070 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:26.528029    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/must-gather-85gd8" event={"ID":"98ab4575-379b-45f5-bc42-9f0676cdccf9","Type":"ContainerStarted","Data":"79a3c534a22425e3ef6bbb451f1501262dea9bbf7d9ec5ec46c4d66c7a4c3d62"}
Nov 01 21:01:26.528600 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:26.528083    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/must-gather-85gd8" event={"ID":"98ab4575-379b-45f5-bc42-9f0676cdccf9","Type":"ContainerStarted","Data":"e9a906d74b95bb33d513cd2b454957064175aac97ca77268f75c414d95147c43"}
Nov 01 21:01:26.546794 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:26.544812    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z9gbm/must-gather-85gd8" podStartSLOduration=1.469104253 podStartE2EDuration="2.544792409s" totalImagesPullingTime="1.075688156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:26.543397252 +0000 UTC m=+3731.959029719" watchObservedRunningTime="2026-11-01 21:01:26.544792409 +0000 UTC m=+3731.960424883"
Nov 01 21:01:27.711320 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:27.711288    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cmf5w_8e0c52c4-326c-4a61-ae5f-b589196d41e6/global-pull-secret-syncer/0.log"
Nov 01 21:01:27.928289 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:27.928252    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log"
Nov 01 21:01:28.067874 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:28.067792    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-52.us-east-2.compute.internal_94566354f58b906840cf934237faff37/haproxy/0.log"
Nov 01 21:01:29.259425 ip-10-0-133-52 kubenswrapper[6479]: E1101 21:01:29.254930    6479 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qczmj" containerName="extractor"
Nov 01 21:01:31.391675 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:31.388277    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 21:01:31.488194 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:31.488139    6479 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jtnpv_0cf8f06e-9873-48b5-8f8a-29b93691922e/konnectivity-agent/0.log" limit=1024
Nov 01 21:01:33.862670 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:33.861783    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-trh22_eff10696-0cfb-4433-ad91-3d0e817fd274/manager/0.log"
Nov 01 21:01:34.895697 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:34.894740    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dvwgc_c59e357a-a3be-4269-9cc3-de751b07d38a/manager/0.log"
Nov 01 21:01:34.943010 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:34.942982    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v5k66_8c6137ad-fb29-4667-be40-173018e00d94/limitador/0.log"
Nov 01 21:01:36.502120 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.502048    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/alertmanager/0.log"
Nov 01 21:01:36.526949 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.526911    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/config-reloader/0.log"
Nov 01 21:01:36.553466 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.553440    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/kube-rbac-proxy-web/0.log"
Nov 01 21:01:36.578992 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.578955    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/kube-rbac-proxy/0.log"
Nov 01 21:01:36.611669 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.610115    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/kube-rbac-proxy-metric/0.log"
Nov 01 21:01:36.661695 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.661074    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/prom-label-proxy/0.log"
Nov 01 21:01:36.685902 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.685870    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aba37058-37a4-48fe-bfc2-12cdca9b241a/init-config-reloader/0.log"
Nov 01 21:01:36.801307 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.801233    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pc6wg_96e9e104-2216-4847-a03b-a6161ac495f2/kube-state-metrics/0.log"
Nov 01 21:01:36.820511 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.820484    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pc6wg_96e9e104-2216-4847-a03b-a6161ac495f2/kube-rbac-proxy-main/0.log"
Nov 01 21:01:36.854815 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.854787    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pc6wg_96e9e104-2216-4847-a03b-a6161ac495f2/kube-rbac-proxy-self/0.log"
Nov 01 21:01:36.883550 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.883514    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b5895c7bb-p9lzh_48922500-8344-4233-92cf-23ad2ff2e844/metrics-server/0.log"
Nov 01 21:01:36.914685 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.914501    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-f4xxd_eea70939-6f2a-485c-8fc9-a335b62860be/monitoring-plugin/0.log"
Nov 01 21:01:36.942727 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.942694    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d5rd4_f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/node-exporter/0.log"
Nov 01 21:01:36.967640 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.967613    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d5rd4_f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/kube-rbac-proxy/0.log"
Nov 01 21:01:36.989458 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:36.989428    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d5rd4_f6da9d8b-2c01-4485-ba36-dbafb7b3e0f2/init-textfile/0.log"
Nov 01 21:01:37.178376 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.178301    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-979sf_4aefdb60-6f0d-4d04-868c-3be0ef759d9d/kube-rbac-proxy-main/0.log"
Nov 01 21:01:37.201395 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.201362    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-979sf_4aefdb60-6f0d-4d04-868c-3be0ef759d9d/kube-rbac-proxy-self/0.log"
Nov 01 21:01:37.225246 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.225206    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-979sf_4aefdb60-6f0d-4d04-868c-3be0ef759d9d/openshift-state-metrics/0.log"
Nov 01 21:01:37.427345 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.427287    6479 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"]
Nov 01 21:01:37.433516 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.433433    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.439007 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.438976    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"]
Nov 01 21:01:37.542533 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.542502    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b8fbbf98d-ddfvf_37d28636-5a23-4fbd-88b3-6d2346c9ed50/telemeter-client/0.log"
Nov 01 21:01:37.551459 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.551427    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-podres\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.551618 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.551504    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-sys\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.551737 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.551614    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-lib-modules\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.551737 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.551717    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-proc\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.551874 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.551823    6479 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7766\" (UniqueName: \"kubernetes.io/projected/6d476c6c-f893-4db9-95a5-1a5d075fbf37-kube-api-access-h7766\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.564402 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.564368    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b8fbbf98d-ddfvf_37d28636-5a23-4fbd-88b3-6d2346c9ed50/reload/0.log"
Nov 01 21:01:37.586671 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.586629    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b8fbbf98d-ddfvf_37d28636-5a23-4fbd-88b3-6d2346c9ed50/kube-rbac-proxy/0.log"
Nov 01 21:01:37.654455 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.654415    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-sys\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.654674 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.654483    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-lib-modules\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.654674 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.654518    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-proc\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.654831 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.654680    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-proc\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.654897 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.654855    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7766\" (UniqueName: \"kubernetes.io/projected/6d476c6c-f893-4db9-95a5-1a5d075fbf37-kube-api-access-h7766\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.654980 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.654964    6479 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-podres\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.655129 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.655113    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-podres\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.655207 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.655173    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-sys\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.655270 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.655247    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d476c6c-f893-4db9-95a5-1a5d075fbf37-lib-modules\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.668460 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.668432    6479 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7766\" (UniqueName: \"kubernetes.io/projected/6d476c6c-f893-4db9-95a5-1a5d075fbf37-kube-api-access-h7766\") pod \"perf-node-gather-daemonset-qb6wn\" (UID: \"6d476c6c-f893-4db9-95a5-1a5d075fbf37\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:37.751043 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:37.751009    6479 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:38.049968 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:38.049893    6479 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"]
Nov 01 21:01:38.053816 ip-10-0-133-52 kubenswrapper[6479]: W1101 21:01:38.053782    6479 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6d476c6c_f893_4db9_95a5_1a5d075fbf37.slice/crio-7bb57b809a5ca41d161721d3c44fcbcd7b8ceb97f69f4b3ccbf5f1cc1473fab5 WatchSource:0}: Error finding container 7bb57b809a5ca41d161721d3c44fcbcd7b8ceb97f69f4b3ccbf5f1cc1473fab5: Status 404 returned error can't find the container with id 7bb57b809a5ca41d161721d3c44fcbcd7b8ceb97f69f4b3ccbf5f1cc1473fab5
Nov 01 21:01:38.230632 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:38.230594    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-trh22_eff10696-0cfb-4433-ad91-3d0e817fd274/manager/0.log"
Nov 01 21:01:38.630582 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:38.629346    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn" event={"ID":"6d476c6c-f893-4db9-95a5-1a5d075fbf37","Type":"ContainerStarted","Data":"ac96f97993ecf5bcfa1cb3159fefebadd0b70ec24cdcbb83ae7567cb8840dc4a"}
Nov 01 21:01:38.630582 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:38.629413    6479 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn" event={"ID":"6d476c6c-f893-4db9-95a5-1a5d075fbf37","Type":"ContainerStarted","Data":"7bb57b809a5ca41d161721d3c44fcbcd7b8ceb97f69f4b3ccbf5f1cc1473fab5"}
Nov 01 21:01:38.630582 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:38.630328    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:38.649675 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:38.649492    6479 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn" podStartSLOduration=1.649475459 podStartE2EDuration="1.649475459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:38.64887755 +0000 UTC m=+3744.064510023" watchObservedRunningTime="2026-11-01 21:01:38.649475459 +0000 UTC m=+3744.065107932"
Nov 01 21:01:42.003482 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:42.003451    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jv26r_baa37f05-fc26-4a6c-a194-e3b4a88a5941/dns/0.log"
Nov 01 21:01:42.024973 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:42.024939    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jv26r_baa37f05-fc26-4a6c-a194-e3b4a88a5941/kube-rbac-proxy/0.log"
Nov 01 21:01:42.049358 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:42.049317    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqvzs_8bf82705-43f4-4c30-a6f3-403bf0f0d899/dns-node-resolver/0.log"
Nov 01 21:01:42.500588 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:42.500547    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-trh22_eff10696-0cfb-4433-ad91-3d0e817fd274/manager/0.log"
Nov 01 21:01:42.752988 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:42.752920    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skvhv_4ae3e006-540b-4bd3-b35e-7c3a9e1d130f/node-ca/0.log"
Nov 01 21:01:43.966712 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:43.966640    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k44gg_c60b32e8-0368-46cc-9385-ba469e82c957/discovery/0.log"
Nov 01 21:01:44.469180 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.469150    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-wwjm6_ff9972fd-5487-4ab7-b0dd-0962e84e88c9/kube-auth-proxy/0.log"
Nov 01 21:01:44.531047 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.531014    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-989zh_4e8677a8-ebe4-4a37-9a4e-e43551a314ff/istio-proxy/0.log"
Nov 01 21:01:44.599716 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.599685    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-c6qrt_c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5/istio-proxy/0.log"
Nov 01 21:01:44.663449 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.663412    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jsxtq_838f2059-c8d7-4b2c-95e9-a9a31c87683c/istio-proxy/0.log"
Nov 01 21:01:44.824026 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.823944    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s4pct_020b4fc0-677d-4ec2-ac2d-bdd1379e7382/istio-proxy/0.log"
Nov 01 21:01:44.873881 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.873848    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7f75799d9b-w7752_d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b/payload-pre-processing/0.log"
Nov 01 21:01:44.902831 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:44.902801    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5ff9ddc8d7-c89gd_7c77aedf-651e-490c-a0e7-31afeacf0102/payload-processing/0.log"
Nov 01 21:01:45.654122 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:45.653188    6479 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-qb6wn"
Nov 01 21:01:46.728992 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:46.728962    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dr9z8_34e81519-4820-4e98-976b-48933a0eb319/serve-healthcheck-canary/0.log"
Nov 01 21:01:47.533004 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.532964    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qczmj_ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/kube-rbac-proxy/0.log"
Nov 01 21:01:47.560150 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.560108    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qczmj_ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/exporter/0.log"
Nov 01 21:01:47.585321 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.585292    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qczmj_ed1f5e20-e8b6-45e3-9d44-1b2f73c3ac3e/extractor/0.log"
Nov 01 21:01:47.694343 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.694314    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k44gg_c60b32e8-0368-46cc-9385-ba469e82c957/discovery/0.log"
Nov 01 21:01:47.704589 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.704561    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-wwjm6_ff9972fd-5487-4ab7-b0dd-0962e84e88c9/kube-auth-proxy/0.log"
Nov 01 21:01:47.740201 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.739047    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-989zh_4e8677a8-ebe4-4a37-9a4e-e43551a314ff/istio-proxy/0.log"
Nov 01 21:01:47.788325 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.788250    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-c6qrt_c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5/istio-proxy/0.log"
Nov 01 21:01:47.827676 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.826805    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jsxtq_838f2059-c8d7-4b2c-95e9-a9a31c87683c/istio-proxy/0.log"
Nov 01 21:01:47.936667 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.935640    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s4pct_020b4fc0-677d-4ec2-ac2d-bdd1379e7382/istio-proxy/0.log"
Nov 01 21:01:47.963629 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.963595    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7f75799d9b-w7752_d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b/payload-pre-processing/0.log"
Nov 01 21:01:47.977664 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:47.977616    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5ff9ddc8d7-c89gd_7c77aedf-651e-490c-a0e7-31afeacf0102/payload-processing/0.log"
Nov 01 21:01:48.227321 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.227282    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k44gg_c60b32e8-0368-46cc-9385-ba469e82c957/discovery/0.log"
Nov 01 21:01:48.251537 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.251484    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-wwjm6_ff9972fd-5487-4ab7-b0dd-0962e84e88c9/kube-auth-proxy/0.log"
Nov 01 21:01:48.291454 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.291425    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-989zh_4e8677a8-ebe4-4a37-9a4e-e43551a314ff/istio-proxy/0.log"
Nov 01 21:01:48.341664 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.341574    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-c6qrt_c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5/istio-proxy/0.log"
Nov 01 21:01:48.383859 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.383828    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jsxtq_838f2059-c8d7-4b2c-95e9-a9a31c87683c/istio-proxy/0.log"
Nov 01 21:01:48.512999 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.512917    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s4pct_020b4fc0-677d-4ec2-ac2d-bdd1379e7382/istio-proxy/0.log"
Nov 01 21:01:48.552689 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.552631    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7f75799d9b-w7752_d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b/payload-pre-processing/0.log"
Nov 01 21:01:48.572301 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:48.572269    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5ff9ddc8d7-c89gd_7c77aedf-651e-490c-a0e7-31afeacf0102/payload-processing/0.log"
Nov 01 21:01:50.392604 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:50.392560    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dvwgc_c59e357a-a3be-4269-9cc3-de751b07d38a/manager/0.log"
Nov 01 21:01:50.404844 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:50.404816    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v5k66_8c6137ad-fb29-4667-be40-173018e00d94/limitador/0.log"
Nov 01 21:01:50.634859 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:50.634831    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-689bcf9f9-kjj7f_5c449c76-8333-45b4-9830-c0147dcd8d35/manager/0.log"
Nov 01 21:01:50.658529 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:50.658442    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-689bcf9f9-kjj7f_5c449c76-8333-45b4-9830-c0147dcd8d35/copy-manifests/0.log"
Nov 01 21:01:51.536092 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.536058    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k44gg_c60b32e8-0368-46cc-9385-ba469e82c957/discovery/0.log"
Nov 01 21:01:51.547219 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.547194    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-wwjm6_ff9972fd-5487-4ab7-b0dd-0962e84e88c9/kube-auth-proxy/0.log"
Nov 01 21:01:51.585198 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.585166    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-989zh_4e8677a8-ebe4-4a37-9a4e-e43551a314ff/istio-proxy/0.log"
Nov 01 21:01:51.623663 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.623622    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-c6qrt_c6f59fb5-3dd5-4c6b-a1fb-94ffb28812a5/istio-proxy/0.log"
Nov 01 21:01:51.658324 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.658295    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jsxtq_838f2059-c8d7-4b2c-95e9-a9a31c87683c/istio-proxy/0.log"
Nov 01 21:01:51.759373 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.759341    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-s4pct_020b4fc0-677d-4ec2-ac2d-bdd1379e7382/istio-proxy/0.log"
Nov 01 21:01:51.789552 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.789470    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7f75799d9b-w7752_d80d2eeb-9b5f-4d37-85bf-c6eb910cdd5b/payload-pre-processing/0.log"
Nov 01 21:01:51.803046 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:51.803022    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5ff9ddc8d7-c89gd_7c77aedf-651e-490c-a0e7-31afeacf0102/payload-processing/0.log"
Nov 01 21:01:52.783486 ip-10-0-133-52 kubenswrapper[6479]: I1101 21:01:52.783446    6479 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-trh22_eff10696-0cfb-4433-ad91-3d0e817fd274/manager/0.log"